Oct 30 12:42:28.780532 ip-10-0-141-232 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 12:42:28.780544 ip-10-0-141-232 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 12:42:28.780551 ip-10-0-141-232 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 12:42:28.780772 ip-10-0-141-232 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 12:42:38.864999 ip-10-0-141-232 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 12:42:38.865020 ip-10-0-141-232 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 077beaf119b24414bc61ddcbbe9d8b7d --
Oct 30 12:44:46.687545 ip-10-0-141-232 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 12:44:47.034121 ip-10-0-141-232 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:44:47.034121 ip-10-0-141-232 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 12:44:47.034121 ip-10-0-141-232 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:44:47.034121 ip-10-0-141-232 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:44:47.038010 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.037924    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:47.038010 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038010    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:47.038010 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038014    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038018    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038021    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038024    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038026    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038029    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038032    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038035    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038037    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038040    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038043    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038046    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038048    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038050    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038053    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038055    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038058    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038061    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038063    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038069    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:47.038109 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038072    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038075    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038077    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038080    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038083    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038086    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038088    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038091    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038094    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038096    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038099    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038101    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038104    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038107    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038110    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038113    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038115    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038118    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038121    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038124    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:47.038602 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038126    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038129    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038137    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038140    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038142    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038145    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038147    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038150    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038153    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038155    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038159    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038162    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038165    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038167    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038171    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038173    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038176    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038178    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038181    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038183    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:47.039073 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038186    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038188    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038191    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038194    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038196    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038200    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038202    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038205    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038207    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038210    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038213    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038215    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038218    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038221    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038223    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038227    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038231    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038234    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038237    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:47.039549 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038240    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038242    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038245    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038249    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038253    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038256    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038259    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038262    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038264    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038267    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038282    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038286    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038289    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038292    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038294    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038297    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038299    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038302    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038305    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:47.039991 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038307    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038310    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038313    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038315    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038318    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.038320    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039118    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039125    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039128    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039131    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039134    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039137    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039140    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039143    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039146    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039148    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039151    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039154    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039157    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:47.040505 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039160    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039163    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039165    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039168    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039170    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039173    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039175    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039178    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039182    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039185    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039188    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039191    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039194    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039196    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039199    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039202    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039204    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039207    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039210    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039212    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:47.040958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039215    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039218    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039220    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039223    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039225    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039228    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039230    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039233    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039236    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039239    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039243    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039245    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039247    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039250    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039253    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039255    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039257    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039260    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039262    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:47.041449 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039265    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039267    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039284    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039287    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039289    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039292    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039295    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039298    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039300    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039303    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039305    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039308    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039311    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039313    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039316    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039318    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039321    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039323    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039325    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039328    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:47.041906 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039330    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039334    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039337    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039340    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039342    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039345    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039347    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039350    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039352    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039355    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039357    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039360    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039362    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039364    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039367    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039369    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039372    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039375    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039378    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039380    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:47.042392 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039382    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039385    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039387    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039390    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039392    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039394    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039397    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039401    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039405    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039407    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039410    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039412    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039414    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039417    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039927    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039932    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039935    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039937    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039940    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:47.042868 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039944    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039947    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039950    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039952    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039955    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039957    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039961    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039965    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039968    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039971    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039974    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039976    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039979    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039981    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039984    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039987    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039989    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039992    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039995    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:47.043326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.039998    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040000    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040004    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040008    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040011    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040013    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040016    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040018    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040021    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040024    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040027    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040030    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040032    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040035    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040037    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040039    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040042    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040044    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040047    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:47.043885 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040050    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040052    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040055    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040058    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040060    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040063    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040065    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040067    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040070    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040072    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040075    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040077    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040083    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040086    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040088    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040090    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040093    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040096    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040098    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:47.044355 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040101    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040103    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040106    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040109    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040111    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040114    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040116    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040119    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040121    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040124    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040126    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040129    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040131    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040133    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040136    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040138    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040141    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040143    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040145    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040148    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:47.044802 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040150    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040153    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040155    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040158    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040160    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040165    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040168    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040170    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040173    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040175    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040178    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040180    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040183    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040185    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040187    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040190    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040194    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040198    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040201    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040203    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:47.045323 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040206    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040208    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040211    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.040214    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041069    2576 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041078    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041087    2576 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041092    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041096    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041099    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041104    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041108    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041111    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041114    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041118    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041121    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041125    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041128    2576 flags.go:64] FLAG: --cgroup-root=""
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041131    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041134    2576 flags.go:64] FLAG: --client-ca-file=""
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041137    2576 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041140    2576 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041147    2576 flags.go:64] FLAG: --cluster-domain=""
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041150    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 12:44:47.045796 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041153    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041156    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041159    2576 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041163    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041166    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041171    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041175    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041177    2576 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041180    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041183    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041187    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041190    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041194    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041197    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041200    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041203    2576 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041206    2576 flags.go:64] FLAG: --enable-server="true"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041209    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041214    2576 flags.go:64] FLAG: --event-burst="100"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041217    2576 flags.go:64] FLAG: --event-qps="50"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041220    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041223    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041226    2576 flags.go:64] FLAG: --eviction-hard=""
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041230    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 12:44:47.046387 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041233    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041236    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041239    2576 flags.go:64] FLAG: --eviction-soft=""
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041242    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041245    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041248    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041251    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041254    2576 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041257    2576 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041260    2576 flags.go:64] FLAG: --feature-gates=""
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041267    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041284    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041287    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041292    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041300    2576 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041303    2576 flags.go:64] FLAG: --help="false"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041307    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041310    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041313    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041315    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041319    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041322    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041325    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 12:44:47.046978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041327    2576 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041330    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041333    2576 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041336    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041339    2576 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041342    2576 flags.go:64] FLAG: --kube-reserved=""
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041345    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041347    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041350    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041353    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041356    2576 flags.go:64] FLAG: --lock-file=""
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041359    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041361    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041364    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041370    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041372    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041375    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041378    2576 flags.go:64] FLAG: --logging-format="text"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041381    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041384    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041387    2576 flags.go:64] FLAG: --manifest-url=""
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041390    2576 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041395    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041398    2576 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041403    2576 flags.go:64] FLAG: --max-pods="110"
Oct 30 12:44:47.047550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041406    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041409    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041412    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041415    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041418    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041421    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041423    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041430    2576 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041433    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041437    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041440    2576 flags.go:64] FLAG: --pod-cidr=""
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041443    2576 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041446    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041449    2576 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041452    2576 flags.go:64] FLAG: --port="10250"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041455    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041458    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-078b9bbb98442f0de"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041462    2576 flags.go:64] FLAG: --qos-reserved=""
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041465    2576 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041468    2576 flags.go:64] FLAG: --register-node="true"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041470    2576 flags.go:64] FLAG: --register-with-taints=""
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041475    2576 flags.go:64] FLAG: --registry-burst="10"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041477    2576 flags.go:64] FLAG: --registry-qps="5"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041480    2576 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041483    2576 flags.go:64] FLAG: --reserved-memory=""
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041487    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 12:44:47.048141 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041490    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041493    2576 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041496    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041500    2576 flags.go:64] FLAG: --runonce="false"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041503    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041506    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041515    2576 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041519    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041522    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041526    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041529    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041532    2576 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041535    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041538    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041540    2576 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041544    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041546    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041550    2576 flags.go:64] FLAG: --system-cgroups=""
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041552    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041558    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041561    2576 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041564    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041570    2576 flags.go:64] FLAG: --tls-min-version=""
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041573    2576 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041576    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 12:44:47.048869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041579    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041581    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041584    2576 flags.go:64] FLAG: --v="2"
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041588    2576 flags.go:64] FLAG: --version="false"
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041592    2576 flags.go:64] FLAG: --vmodule=""
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041596    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.041599    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041753    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041757    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041760    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041763    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041766    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041768    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041771    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041775    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041778    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041781    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041784    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041786    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041789    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041791    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:47.049466 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041794    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041796    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041799    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041801    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041804    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041806    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041809    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041812    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041815    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041817    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041820    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041822    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041825    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041827    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041830    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041832    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041835    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041837    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041840    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041842    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:47.049959 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041845    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041849    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041851    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041855    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041859    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041866    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041870    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041873    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041876    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041878    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041881    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041884    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041886    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041889    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041891    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041894    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041896    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041899    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041902    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041904    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:47.050470 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041907    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041909    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041912    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041914    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041917    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041920    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041922    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041924    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041927    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041929    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041932    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041934    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041937    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041940    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041943    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041945    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041947    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041951    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041954    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:47.050937 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041957    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041959    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041962    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041964    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041966    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041969    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041971    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041973    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041976    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041979    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041981    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041984    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041986    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041988    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041991    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041993    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041995    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.041998    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042000    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:47.051391 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042003    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042005    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042008    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042010    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042014    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042017    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042019    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042022    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042024    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042027    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042029    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042033    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042035    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042038    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.042065    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042099    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:47.051844 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042103    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042106    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042109    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042112    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042114    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042117    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042119    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042122    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042126    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042130    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042132    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042135    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042137    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042140    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042143    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042145    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042148    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042151    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042153    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042156    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:47.052225 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042158    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042161    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042163    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042165    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042168    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042170    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042173    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042177    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042179    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042182    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042184    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042187    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042190    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042192    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042194    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042197    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042199    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042202    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042204    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:47.052788 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042207    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042209    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042212    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042214    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042216    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042219    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042221    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042224    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042226    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042228    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042231    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042233    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042235    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042238    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042240    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042243    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042245    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042248    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042250    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:47.053241 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042253    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042256    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042259    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042261    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042263    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042266    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042282    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042285    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042288    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042292    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042295    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042299    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042302    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042304    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042307    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042310    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042312    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042314    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042317    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:47.053698 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042319    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042323    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042325    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042328    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042330    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042333    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042335    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042338    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042341    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042343    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042346    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042350    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042352    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042355    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042359    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042361    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042364    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042367    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042369    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:47.054168 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042372    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042375    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042378    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042381    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042383    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042386    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042389    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042391    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.042394    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.049413    2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.049430    2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.049446    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.049451    2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049507    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049512    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049516    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049519    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049522    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049524    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049528    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:47.054629 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049533    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049537    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049540    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049542    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049545    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049547    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049550    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049553    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049555    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049557    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049560    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049562    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049565    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049568    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049570    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049573    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049576    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049579    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049582    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:47.055208 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049584    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049587    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049589    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049593    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049596    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049617    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049621    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049625    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049628    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049631    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049634    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049636    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049639    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049642    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049645    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049647    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049650    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049652    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049655    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049657    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:47.055673 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049660    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049663    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049665    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049667    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049670    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049672    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049675    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049677    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049680    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049684    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049687    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049690    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049693    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049696    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049699    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049703    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049706    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049708    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049711    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:47.056142 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049714    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049716    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049718    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049721    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049724    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049726    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049729    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049731    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049733    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049736    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049738    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049741    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049743    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049746    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049748    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049750    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049753    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049755    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049758    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049760    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:47.056605 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049762    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049765    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049768    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049771    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049773    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049776    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049778    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049782    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049784    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049787    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049790    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049793    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049795    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049798    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049800    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049802    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049805    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049807    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049810    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:47.057070 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049812    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049815    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.049843    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049994    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.049998    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050002    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050005    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050007    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050010    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050012    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050015    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050017    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050020    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050022    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050025    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050028    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:47.057533 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050030    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050032    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050035    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050037    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050040    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050043    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050046    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050048    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050051    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050053    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050056    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050058    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050060    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050063    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050065    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050068    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050070    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050072    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050075    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:47.057958 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050077    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050080    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050082    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050085    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050087    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050090    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050092    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050095    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050097    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050101    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050104    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050107    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050110    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050113    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050116    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050118    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050121    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050123    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050126    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:47.058634 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050128    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050131    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050133    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050136    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050138    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050141    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050143    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050146    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050148    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050151    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050153    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050155    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050158    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050160    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050162    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050165    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050167    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050170    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050172    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050174    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:47.059084 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050177    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050179    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050182    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050185    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050187    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050189    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050192    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050194    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050197    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050199    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050201    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050204    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050206    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050208    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050211    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050213    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050217    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050219    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050221    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050224    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:47.059783 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050226    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050229    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050231    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050234    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050236    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050239    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050241    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050244    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050246    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050249    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050251    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050253    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050260    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050263    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050267    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:47.061120 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.050304    2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050336    2576 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050340    2576 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050344    2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050346    2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050349    2576 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050352    2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050354    2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050357    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050359    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050362    2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050364    2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050367    2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050371    2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050375    2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050378    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050381    2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050383    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050386    2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:44:47.061557 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050388    2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050391    2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050393    2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050396    2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050398    2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050401    2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050411    2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050414    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050417    2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050419    2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050422    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050426    2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050429    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050431    2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050434    2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050436    2576 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050439    2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050442    2576 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050444    2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050447    2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:44:47.062048 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050449    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050452    2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050454    2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050457    2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050459    2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050462    2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050464    2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050467    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050470    2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050473    2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050476    2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050479    2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050482    2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050485    2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050487    2576 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050490    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050493    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050495    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050498    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050500    2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:44:47.062628 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050503    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050505    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050508    2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050512    2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050514    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050517    2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050520    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050522    2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050524    2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050527    2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050530    2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050532    2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050535    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050538    2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050540    2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050544    2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050547    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050550    2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050553    2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:44:47.063228 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050555    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050558    2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050561    2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050563    2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050566    2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050568    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050571    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050573    2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050575    2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050578    2576 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050581    2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050583    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050585    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050588    2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050590    2576 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050593    2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050596    2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050599    2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050602    2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050605    2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:44:47.063969 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050607    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050610    2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050613    2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050615    2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050618    2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050620    2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050622    2576 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050625    2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:47.050627    2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.050681    2576 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.052800    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.053630    2576 server.go:1014] "Starting client certificate rotation"
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.053725    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.053766    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.055461    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:44:47.064608 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.059353    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:44:47.068248 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.068231    2576 log.go:25] "Validated CRI v1 runtime API"
Oct 30 12:44:47.071515 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.071500    2576 log.go:25] "Validated CRI v1 image API"
Oct 30 12:44:47.072324 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.072295    2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 12:44:47.074661 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.074642    2576 fs.go:135] Filesystem UUIDs: map[76dcbf4e-7fa3-485c-b477-0cf87d7bd7cb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 89e1d28a-48d5-44c6-ac23-7c268b390908:/dev/nvme0n1p4]
Oct 30 12:44:47.074706 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.074661    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 12:44:47.080367 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.080245    2576 manager.go:217] Machine: {Timestamp:2026-10-30 12:44:47.078991761 +0000 UTC m=+0.291286282 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3083544 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a6b771ad6894acb540105234eb7f SystemUUID:ec28a6b7-71ad-6894-acb5-40105234eb7f BootID:077beaf1-19b2-4414-bc61-ddcbbe9d8b7d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:db:92:f4:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:db:92:f4:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:4f:bc:6c:e1:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 12:44:47.080925 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.080916    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 12:44:47.081008 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.080996    2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 12:44:47.083544 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.083520    2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 12:44:47.083700 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.083547    2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-232.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 12:44:47.083744 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.083704    2576 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 12:44:47.083744 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.083712    2576 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 12:44:47.083792 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.083752    2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:44:47.083792 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.083767    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:44:47.084850 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.084839    2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 12:44:47.088943 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.088931    2576 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 12:44:47.089070 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.089060    2576 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 12:44:47.089543 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.089533    2576 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 12:44:47.089587 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.089546    2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 12:44:47.089587 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.089560    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 12:44:47.089587 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.089569    2576 kubelet.go:405] "Adding apiserver pod source"
Oct 30 12:44:47.089587 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.089579    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 12:44:47.089878 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.089864    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:44:47.091756 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.091737    2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 12:44:47.098260 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.098247    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:44:47.098955 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.098943    2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 12:44:47.098990 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.098982    2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 12:44:47.099144 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.099134    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 12:44:47.099177 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.099154    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 12:44:47.099177 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.099164    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 12:44:47.099177 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.099173    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 12:44:47.099250 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.099182    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 12:44:47.099250 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.099190    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 12:44:47.099792 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.099782    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 12:44:47.099820 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.099793    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 12:44:47.099820 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.099814    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 12:44:47.099820 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.099820    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 12:44:47.099905 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.099832    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 12:44:47.099905 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.099841    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 12:44:47.100399 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.100390    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 12:44:47.100428 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.100399    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 12:44:47.104040 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.104028    2576 server.go:1263] "Started kubelet"
Oct 30 12:44:47.104151 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.104103    2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 12:44:47.104233 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.104198    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 12:44:47.104295 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.104261    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 12:44:47.107092 ip-10-0-141-232 systemd[1]: Started Kubernetes Kubelet.
Oct 30 12:44:47.108722 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.108653    2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 12:44:47.111033 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.111013    2576 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 12:44:47.112160 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.112138    2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 12:44:47.116227 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.116210    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 12:44:47.117254 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.117236    2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 12:44:47.117374 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.117351    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:47.117809 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.117792    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 12:44:47.117856 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.117794    2576 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 12:44:47.117856 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.117824    2576 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 12:44:47.117941 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.117931    2576 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 12:44:47.117977 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.117941    2576 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 12:44:47.119372 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.119355    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 12:44:47.119372 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.119371    2576 factory.go:55] Registering systemd factory
Oct 30 12:44:47.119510 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.119378    2576 factory.go:223] Registration of the systemd container factory successfully
Oct 30 12:44:47.119610 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.119597    2576 factory.go:153] Registering CRI-O factory
Oct 30 12:44:47.119657 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.119615    2576 factory.go:223] Registration of the crio container factory successfully
Oct 30 12:44:47.119657 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.119633    2576 factory.go:103] Registering Raw factory
Oct 30 12:44:47.119657 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.119646    2576 manager.go:1199] Started watching for new ooms in manager
Oct 30 12:44:47.120584 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.120571    2576 manager.go:319] Starting recovery of all containers
Oct 30 12:44:47.131181 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.131165    2576 manager.go:324] Recovery completed
Oct 30 12:44:47.135052 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.135035    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:47.136649 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.136621    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:47.136721 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.136671    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:47.136721 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.136683    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:47.137565 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.137527    2576 cpu_manager.go:226] "Starting" policy="none"
Oct 30 12:44:47.137640 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.137566    2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 12:44:47.137640 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.137621    2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 12:44:47.137759 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.137553    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:47.138759 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.138719    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:47.138848 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.138780    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:47.138848 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.138795    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:47.140198 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.140179    2576 policy_none.go:50] "Start"
Oct 30 12:44:47.140266 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.140208    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 12:44:47.140266 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.140221    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 12:44:47.142216 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.142200    2576 policy_none.go:44] "Start"
Oct 30 12:44:47.142452 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.142425    2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 12:44:47.142452 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.142419    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-232.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 12:44:47.155612 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.154487    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-232.ec2.internal.18e34f5145b4a58d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-232.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-232.ec2.internal,},FirstTimestamp:2026-10-30 12:44:47.104001421 +0000 UTC m=+0.316295941,LastTimestamp:2026-10-30 12:44:47.104001421 +0000 UTC m=+0.316295941,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-232.ec2.internal,}"
Oct 30 12:44:47.155721 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.155678    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 12:44:47.155797 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.155743    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mndrm"
Oct 30 12:44:47.167163 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.167141    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 12:44:47.167163 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.167141    2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-232.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 12:44:47.167973 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.167954    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mndrm"
Oct 30 12:44:47.170849 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.170821    2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 12:44:47.173125 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.173113    2576 manager.go:341] "Starting Device Plugin manager"
Oct 30 12:44:47.173190 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.173143    2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 12:44:47.173190 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.173155    2576 server.go:87] "Starting device plugin registration server"
Oct 30 12:44:47.173401 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.173390    2576 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 12:44:47.173441 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.173402    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 12:44:47.173533 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.173514    2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 12:44:47.173617 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.173604    2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 12:44:47.173617 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.173617    2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 12:44:47.174112 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.174094    2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 12:44:47.174199 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.174145    2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:47.228716 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.222425    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:44:47.253906 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.253871    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 12:44:47.255503 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.255483    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 12:44:47.255503 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.255505    2576 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 12:44:47.255814 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.255528    2576 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 12:44:47.255814 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.255558    2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 12:44:47.269789 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.269763    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:44:47.274084 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.274070    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:47.274929 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.274912    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:47.275027 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.274946    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:47.275027 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.274961    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:47.275027 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.274992    2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.293693 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.293647    2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.293693 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.293671    2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-232.ec2.internal\": node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:47.331325 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.331288    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:47.356551 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.356520    2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"]
Oct 30 12:44:47.356620 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.356589    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:47.357588 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.357574    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:47.357649 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.357603    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:47.357649 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.357614    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:47.359408 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.359395    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:47.359584 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.359571    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.359623 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.359601    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:47.360114 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.360097    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:47.360187 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.360126    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:47.360187 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.360136    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:47.360187 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.360162    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:47.360187 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.360176    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:47.360389 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.360141    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:47.362028 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.362012    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.362108 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.362043    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:47.362781 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.362767    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:47.362858 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.362792    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:47.362858 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.362801    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:47.386290 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.386260    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-232.ec2.internal\" not found" node="ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.388960 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.388945    2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-232.ec2.internal\" not found" node="ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.420326 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.420303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6886c36025c26ee79576270d5200a014-config\") pod \"kube-apiserver-proxy-ip-10-0-141-232.ec2.internal\" (UID: \"6886c36025c26ee79576270d5200a014\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.420420 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.420332    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8bd9c92b47bd47faf7dc12d1bdd0a47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"e8bd9c92b47bd47faf7dc12d1bdd0a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.420420 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.420356    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8bd9c92b47bd47faf7dc12d1bdd0a47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"e8bd9c92b47bd47faf7dc12d1bdd0a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.439240 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.432610    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:47.521484 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.521454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6886c36025c26ee79576270d5200a014-config\") pod \"kube-apiserver-proxy-ip-10-0-141-232.ec2.internal\" (UID: \"6886c36025c26ee79576270d5200a014\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.521484 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.521482    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6886c36025c26ee79576270d5200a014-config\") pod \"kube-apiserver-proxy-ip-10-0-141-232.ec2.internal\" (UID: \"6886c36025c26ee79576270d5200a014\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.521614 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.521525    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8bd9c92b47bd47faf7dc12d1bdd0a47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"e8bd9c92b47bd47faf7dc12d1bdd0a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.521614 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.521545    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8bd9c92b47bd47faf7dc12d1bdd0a47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"e8bd9c92b47bd47faf7dc12d1bdd0a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.521614 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.521594    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8bd9c92b47bd47faf7dc12d1bdd0a47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"e8bd9c92b47bd47faf7dc12d1bdd0a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.521614 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.521597    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8bd9c92b47bd47faf7dc12d1bdd0a47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"e8bd9c92b47bd47faf7dc12d1bdd0a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.532878 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.532857    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:47.633232 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.633173    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:47.691661 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.691634    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.693795 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:47.693774    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:47.734265 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.734231    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:47.834787 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.834740    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:47.935305 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:47.935214    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:48.035716 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:48.035685    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:48.053887 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.053856    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 12:44:48.054039 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.054020    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 12:44:48.054077 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.054039    2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 12:44:48.136795 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:48.136764    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:48.173589 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.173530    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:39:47 +0000 UTC" deadline="2028-07-05 13:33:02.451889033 +0000 UTC"
Oct 30 12:44:48.173589 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.173582    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14736h48m14.278309527s"
Oct 30 12:44:48.237962 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:48.237886    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:48.272796 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:48.272767    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bd9c92b47bd47faf7dc12d1bdd0a47.slice/crio-a1bdce8d58370fdadfaf068b0920e59bb20f3e3f7d66e53d3a6646e9a2d8753e WatchSource:0}: Error finding container a1bdce8d58370fdadfaf068b0920e59bb20f3e3f7d66e53d3a6646e9a2d8753e: Status 404 returned error can't find the container with id a1bdce8d58370fdadfaf068b0920e59bb20f3e3f7d66e53d3a6646e9a2d8753e
Oct 30 12:44:48.273359 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:48.273343    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6886c36025c26ee79576270d5200a014.slice/crio-86db01e788f2fc0e7e4e20864659d62fbe73c296d6217476e2cbb5065b738b3a WatchSource:0}: Error finding container 86db01e788f2fc0e7e4e20864659d62fbe73c296d6217476e2cbb5065b738b3a: Status 404 returned error can't find the container with id 86db01e788f2fc0e7e4e20864659d62fbe73c296d6217476e2cbb5065b738b3a
Oct 30 12:44:48.278985 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.278964    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:44:48.338790 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:48.338737    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:48.341906 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.341882    2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:44:48.343113 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.343095    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:44:48.343194 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.343125    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:44:48.343194 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.343139    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:44:48.439591 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:48.439561    2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:44:48.471805 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.471779    2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:44:48.517875 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.517812    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:48.539135 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.539115    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:44:48.539250 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.539227    2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Oct 30 12:44:48.549503 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.549481    2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:44:48.557610 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.557595    2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:44:48.926968 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:48.926892    2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:44:49.090302 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.090254    2576 apiserver.go:52] "Watching apiserver"
Oct 30 12:44:49.117894 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.117863    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 12:44:49.123077 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.123057    2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 12:44:49.123437 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.123415    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal","openshift-multus/multus-additional-cni-plugins-jzgc2","openshift-multus/multus-x48w4","kube-system/konnectivity-agent-fd9xb","openshift-dns/node-resolver-tg5ns","openshift-image-registry/node-ca-bzkw4","openshift-network-operator/iptables-alerter-jtfpn","openshift-ovn-kubernetes/ovnkube-node-dg4xp","kube-system/global-pull-secret-syncer-79f2f","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal","openshift-multus/network-metrics-daemon-zfp4t","openshift-network-diagnostics/network-check-target-7qcht","openshift-cluster-node-tuning-operator/tuned-gxdv2"]
Oct 30 12:44:49.126124 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.126108    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:44:49.127908 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.127885    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:44:49.127999 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.127946    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:44:49.129916 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.129897    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.132016 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.132001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:44:49.134169 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.134147    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:44:49.135964 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.135774    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:44:49.137847 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.137829    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:49.137931 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.137896    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:44:49.140471 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.139901    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.142644 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.142624    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.145251 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.145219    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.146772 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.146753    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6g9fh\""
Oct 30 12:44:49.146845 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.146774    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 12:44:49.146905 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.146891    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 12:44:49.148367 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.148134    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b79n8\""
Oct 30 12:44:49.148367 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.148183    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l246t\""
Oct 30 12:44:49.148367 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.148349    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 12:44:49.148630 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.148612    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 12:44:49.148855 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.148842    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 12:44:49.149010 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.148999    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:44:49.149220 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.149193    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 12:44:49.149491 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.149469    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.149576 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.149518    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 12:44:49.149799 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.149785    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 12:44:49.150869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.150189    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 12:44:49.150869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.148454    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 12:44:49.150869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.150785    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 12:44:49.151402 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.151382    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-trr9l\""
Oct 30 12:44:49.153540 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.153523    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:49.153641 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.153608    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:44:49.163349 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.163323    2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 12:44:49.168113 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.168077    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cgnj4"
Oct 30 12:44:49.177597 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.177527    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cgnj4"
Oct 30 12:44:49.219656 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.219634    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 12:44:49.232025 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.231994    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8053a33-e1bf-43f4-b23b-34694e7ce042-host-slash\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:44:49.232192 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-cni-binary-copy\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.232192 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232055    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:49.232192 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.232192 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.232192 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232131    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.232192 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232153    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86mph\" (UniqueName: \"kubernetes.io/projected/52f5c091-40a2-46e8-9214-ddf02da33d0b-kube-api-access-86mph\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:44:49.232192 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232176    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-systemd\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-run\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232221    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-sys-fs\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232250    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9whn4\" (UniqueName: \"kubernetes.io/projected/36c3843d-81b7-4554-80ab-f6fc5901f488-kube-api-access-9whn4\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232290    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8053a33-e1bf-43f4-b23b-34694e7ce042-iptables-alerter-script\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-system-cni-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232337    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-sys\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232393    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-run-netns\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232414    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-cni-bin\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-ovnkube-script-lib\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232462    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-multus-certs\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232483    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skwdd\" (UniqueName: \"kubernetes.io/projected/ba6439a7-7042-4edf-aa9c-6d33b168873d-kube-api-access-skwdd\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-systemd\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232525    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-cni-netd\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.232549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232552    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-cni-multus\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52f5c091-40a2-46e8-9214-ddf02da33d0b-hosts-file\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232593    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-host\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232613    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-node-log\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-kubelet\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysconfig\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232686    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-lib-modules\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba6439a7-7042-4edf-aa9c-6d33b168873d-tmp\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-kubelet\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232767    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e-agent-certs\") pod \"konnectivity-agent-fd9xb\" (UID: \"d08a5487-7eaa-4438-8fcd-3ead8dc6c20e\") " pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dccd197b-ec88-445b-8de9-ff83832112bd-serviceca\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232811    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-env-overrides\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-socket-dir-parent\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232861    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-hostroot\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-etc-kubernetes\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e-konnectivity-ca\") pod \"konnectivity-agent-fd9xb\" (UID: \"d08a5487-7eaa-4438-8fcd-3ead8dc6c20e\") " pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232933    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-ovn\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.233121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-ovnkube-config\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.232978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq4cv\" (UniqueName: \"kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv\") pod \"network-check-target-7qcht\" (UID: \"34fc63c2-b395-4702-a2e4-134161c0d50c\") " pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-cni-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-k8s-cni-cncf-io\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233048    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-netns\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233070    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlm6t\" (UniqueName: \"kubernetes.io/projected/dccd197b-ec88-445b-8de9-ff83832112bd-kube-api-access-rlm6t\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233091    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-var-lib-kubelet\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233145    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-registration-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233168    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-slash\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-var-lib-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233222    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/408c5c75-c039-4dc1-89c7-6b0f4412aa59-dbus\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233244    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-cnibin\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233288    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233311    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-conf-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8s98\" (UniqueName: \"kubernetes.io/projected/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-kube-api-access-x8s98\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.233761 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233493    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysctl-d\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233602    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm9qn\" (UniqueName: \"kubernetes.io/projected/a8053a33-e1bf-43f4-b23b-34694e7ce042-kube-api-access-sm9qn\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-cnibin\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233675    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-systemd-units\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-log-socket\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233730    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh9mp\" (UniqueName: \"kubernetes.io/projected/f6897566-da42-4b8d-ba85-7ec304624f2b-kube-api-access-dh9mp\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233785    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-cni-bin\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-daemon-config\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-modprobe-d\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysctl-conf\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233950    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm2ml\" (UniqueName: \"kubernetes.io/projected/429e3a90-1868-4632-8631-5c0207978063-kube-api-access-wm2ml\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.233978    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.234003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f5c091-40a2-46e8-9214-ddf02da33d0b-tmp-dir\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.234073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-tuned\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.234104    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-system-cni-dir\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.234384 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.234124    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-cni-binary-copy\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.234919 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.234157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-socket-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.234919 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.234175    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.234919 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.234202    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dccd197b-ec88-445b-8de9-ff83832112bd-host\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:44:49.234919 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.234263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-etc-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.234919 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.234307    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6897566-da42-4b8d-ba85-7ec304624f2b-ovn-node-metrics-cert\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.234919 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.234329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/408c5c75-c039-4dc1-89c7-6b0f4412aa59-kubelet-config\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:49.234919 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.234358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-kubernetes\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.234919 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.234377    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-device-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.234919 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.234398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcxs7\" (UniqueName: \"kubernetes.io/projected/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-kube-api-access-lcxs7\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:49.238580 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238347    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 12:44:49.238580 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238455    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 12:44:49.238580 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238470    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 12:44:49.238580 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238517    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 12:44:49.238580 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238526    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 12:44:49.238580 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238533    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 12:44:49.238580 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238552    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 12:44:49.238944 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238718    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 12:44:49.238944 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238729    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:44:49.238944 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238785    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 12:44:49.238944 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238790    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 12:44:49.238944 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238802    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 12:44:49.238944 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238848    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6f4z2\""
Oct 30 12:44:49.238944 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238855    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-svz4t\""
Oct 30 12:44:49.238944 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238856    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 12:44:49.238944 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238887    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 12:44:49.238944 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238945    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pglr2\""
Oct 30 12:44:49.239307 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238957    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mwttw\""
Oct 30 12:44:49.239307 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.238959    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k9vxw\""
Oct 30 12:44:49.239307 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.239014    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 12:44:49.246329 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.246246    2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:44:49.261234 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.261187    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal" event={"ID":"6886c36025c26ee79576270d5200a014","Type":"ContainerStarted","Data":"86db01e788f2fc0e7e4e20864659d62fbe73c296d6217476e2cbb5065b738b3a"}
Oct 30 12:44:49.262453 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.262432    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal" event={"ID":"e8bd9c92b47bd47faf7dc12d1bdd0a47","Type":"ContainerStarted","Data":"a1bdce8d58370fdadfaf068b0920e59bb20f3e3f7d66e53d3a6646e9a2d8753e"}
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-hostroot\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334676    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-etc-kubernetes\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334723    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e-konnectivity-ca\") pod \"konnectivity-agent-fd9xb\" (UID: \"d08a5487-7eaa-4438-8fcd-3ead8dc6c20e\") " pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334732    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-etc-kubernetes\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-ovn\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-ovnkube-config\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334789    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-hostroot\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4cv\" (UniqueName: \"kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv\") pod \"network-check-target-7qcht\" (UID: \"34fc63c2-b395-4702-a2e4-134161c0d50c\") " pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-cni-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334837    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-ovn\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-k8s-cni-cncf-io\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-netns\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334905    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlm6t\" (UniqueName: \"kubernetes.io/projected/dccd197b-ec88-445b-8de9-ff83832112bd-kube-api-access-rlm6t\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334933    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-var-lib-kubelet\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.334991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-registration-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-slash\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.336920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335026    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-var-lib-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335069    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/408c5c75-c039-4dc1-89c7-6b0f4412aa59-dbus\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335083    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-cni-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335091    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-cnibin\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335121    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335143    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-conf-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335180    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8s98\" (UniqueName: \"kubernetes.io/projected/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-kube-api-access-x8s98\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335231    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysctl-d\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm9qn\" (UniqueName: \"kubernetes.io/projected/a8053a33-e1bf-43f4-b23b-34694e7ce042-kube-api-access-sm9qn\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-cnibin\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-systemd-units\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-log-socket\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9mp\" (UniqueName: \"kubernetes.io/projected/f6897566-da42-4b8d-ba85-7ec304624f2b-kube-api-access-dh9mp\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335515    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-cni-bin\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-daemon-config\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335558    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/408c5c75-c039-4dc1-89c7-6b0f4412aa59-dbus\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:49.337783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-modprobe-d\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysctl-conf\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335619    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-ovnkube-config\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm2ml\" (UniqueName: \"kubernetes.io/projected/429e3a90-1868-4632-8631-5c0207978063-kube-api-access-wm2ml\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335678    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335711    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-modprobe-d\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f5c091-40a2-46e8-9214-ddf02da33d0b-tmp-dir\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335738    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335747    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-tuned\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335769    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-system-cni-dir\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-cni-binary-copy\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335815    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-socket-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dccd197b-ec88-445b-8de9-ff83832112bd-host\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335907    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-etc-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335935    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6897566-da42-4b8d-ba85-7ec304624f2b-ovn-node-metrics-cert\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/408c5c75-c039-4dc1-89c7-6b0f4412aa59-kubelet-config\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:49.338470 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-kubernetes\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.335997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-device-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336013    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f5c091-40a2-46e8-9214-ddf02da33d0b-tmp-dir\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336018    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcxs7\" (UniqueName: \"kubernetes.io/projected/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-kube-api-access-lcxs7\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8053a33-e1bf-43f4-b23b-34694e7ce042-host-slash\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336093    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-cni-binary-copy\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336120    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336176    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e-konnectivity-ca\") pod \"konnectivity-agent-fd9xb\" (UID: \"d08a5487-7eaa-4438-8fcd-3ead8dc6c20e\") " pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336240    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-k8s-cni-cncf-io\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336292    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-netns\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336310    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336367    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86mph\" (UniqueName: \"kubernetes.io/projected/52f5c091-40a2-46e8-9214-ddf02da33d0b-kube-api-access-86mph\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336362    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-cnibin\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-systemd\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.339160 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-run\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-sys-fs\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336541    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-var-lib-kubelet\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9whn4\" (UniqueName: \"kubernetes.io/projected/36c3843d-81b7-4554-80ab-f6fc5901f488-kube-api-access-9whn4\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336585    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8053a33-e1bf-43f4-b23b-34694e7ce042-iptables-alerter-script\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-system-cni-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-sys\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336676    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336698    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-run-netns\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-cni-bin\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-ovnkube-script-lib\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336808    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-multus-certs\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skwdd\" (UniqueName: \"kubernetes.io/projected/ba6439a7-7042-4edf-aa9c-6d33b168873d-kube-api-access-skwdd\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.336880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-systemd\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.337341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-cni-netd\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.337383    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-cni-multus\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.337416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52f5c091-40a2-46e8-9214-ddf02da33d0b-hosts-file\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:44:49.339841 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.337505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-host\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.337538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-node-log\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.337569    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-kubelet\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.337615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysconfig\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.337665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-lib-modules\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.337694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba6439a7-7042-4edf-aa9c-6d33b168873d-tmp\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.337790    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-kubelet\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.337857    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-registration-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.337932    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-slash\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.337979    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-var-lib-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.338113    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysctl-conf\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.338494    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.338555    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8053a33-e1bf-43f4-b23b-34694e7ce042-host-slash\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.340005    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-daemon-config\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.340132    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.340532 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.340489    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-kubelet\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.341102 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.341039    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-cni-binary-copy\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.341146 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.341132    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:49.341460 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.341213    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret podName:408c5c75-c039-4dc1-89c7-6b0f4412aa59 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:49.841170295 +0000 UTC m=+3.053464828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret") pod "global-pull-secret-syncer-79f2f" (UID: "408c5c75-c039-4dc1-89c7-6b0f4412aa59") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.341476    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.341675    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-systemd\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.341754    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-run\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.342184    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.342320    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-system-cni-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.342326    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-sys\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.342474    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-run-netns\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.342511    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-cni-bin\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.342542    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.342680    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8053a33-e1bf-43f4-b23b-34694e7ce042-iptables-alerter-script\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.342708    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-multus-certs\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.342995    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-systemd\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.343048    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-cni-netd\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.343097    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-cni-multus\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.343143    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-ovnkube-script-lib\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.343157    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52f5c091-40a2-46e8-9214-ddf02da33d0b-hosts-file\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.343209    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-host\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.343572 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.343259    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-node-log\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.344407 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.343351    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-kubelet\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.344407 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.343425    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysconfig\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.344407 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.343631    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dccd197b-ec88-445b-8de9-ff83832112bd-host\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:44:49.344407 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.343828    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-system-cni-dir\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.344407 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.343839    2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 12:44:49.344407 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.344054    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-socket-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.344407 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.344154    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.344407 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.344175    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/408c5c75-c039-4dc1-89c7-6b0f4412aa59-kubelet-config\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:49.344407 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.344333    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-device-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.344797 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.344417    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-kubernetes\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.344797 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.344423    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-cni-binary-copy\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.344797 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.344487    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-conf-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.344797 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.344529    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-cnibin\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.344797 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.344562    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-etc-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.344797 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.344794    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysctl-d\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.345053 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.344814    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:49.345053 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.344891    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs podName:825a2efb-6a8c-4bdd-bf89-cca7ee710f8e nodeName:}" failed. No retries permitted until 2026-10-30 12:44:49.844861823 +0000 UTC m=+3.057156343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs") pod "network-metrics-daemon-zfp4t" (UID: "825a2efb-6a8c-4bdd-bf89-cca7ee710f8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:49.345053 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.345004    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-log-socket\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.345206 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.345050    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-lib-modules\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.345206 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.345091    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-cni-bin\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.345206 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.345144    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e-agent-certs\") pod \"konnectivity-agent-fd9xb\" (UID: \"d08a5487-7eaa-4438-8fcd-3ead8dc6c20e\") " pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:44:49.345206 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.345163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dccd197b-ec88-445b-8de9-ff83832112bd-serviceca\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:44:49.345206 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.345186    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-env-overrides\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.345451 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.345236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-socket-dir-parent\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.345451 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.345320    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-systemd-units\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.345451 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.345396    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-socket-dir-parent\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.345682 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.345597    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-sys-fs\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.346564 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.346541    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dccd197b-ec88-445b-8de9-ff83832112bd-serviceca\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:44:49.346704 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.346674    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-env-overrides\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.347800 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.347742    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-tuned\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.347800 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.347783    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6897566-da42-4b8d-ba85-7ec304624f2b-ovn-node-metrics-cert\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.348418 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.348398    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba6439a7-7042-4edf-aa9c-6d33b168873d-tmp\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.348606 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.348584    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e-agent-certs\") pod \"konnectivity-agent-fd9xb\" (UID: \"d08a5487-7eaa-4438-8fcd-3ead8dc6c20e\") " pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:44:49.353526 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.353506    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:44:49.353613 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.353530    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:44:49.353613 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.353556    2576 projected.go:196] Error preparing data for projected volume kube-api-access-gq4cv for pod openshift-network-diagnostics/network-check-target-7qcht: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:49.353712 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.353612    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv podName:34fc63c2-b395-4702-a2e4-134161c0d50c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:49.853596606 +0000 UTC m=+3.065891135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gq4cv" (UniqueName: "kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv") pod "network-check-target-7qcht" (UID: "34fc63c2-b395-4702-a2e4-134161c0d50c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:49.356485 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.356462    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcxs7\" (UniqueName: \"kubernetes.io/projected/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-kube-api-access-lcxs7\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:49.357162 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.356833    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlm6t\" (UniqueName: \"kubernetes.io/projected/dccd197b-ec88-445b-8de9-ff83832112bd-kube-api-access-rlm6t\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:44:49.358758 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.358732    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm2ml\" (UniqueName: \"kubernetes.io/projected/429e3a90-1868-4632-8631-5c0207978063-kube-api-access-wm2ml\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.364245 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.364202    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86mph\" (UniqueName: \"kubernetes.io/projected/52f5c091-40a2-46e8-9214-ddf02da33d0b-kube-api-access-86mph\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:44:49.364816 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.364789    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9whn4\" (UniqueName: \"kubernetes.io/projected/36c3843d-81b7-4554-80ab-f6fc5901f488-kube-api-access-9whn4\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.366765 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.366741    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm9qn\" (UniqueName: \"kubernetes.io/projected/a8053a33-e1bf-43f4-b23b-34694e7ce042-kube-api-access-sm9qn\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:44:49.366765 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.366755    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8s98\" (UniqueName: \"kubernetes.io/projected/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-kube-api-access-x8s98\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.366765 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.366761    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skwdd\" (UniqueName: \"kubernetes.io/projected/ba6439a7-7042-4edf-aa9c-6d33b168873d-kube-api-access-skwdd\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.368825 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.368788    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh9mp\" (UniqueName: \"kubernetes.io/projected/f6897566-da42-4b8d-ba85-7ec304624f2b-kube-api-access-dh9mp\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.441420 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.441349    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:44:49.448373 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.448348    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-x48w4"
Oct 30 12:44:49.454918 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:49.454893    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8053a33_e1bf_43f4_b23b_34694e7ce042.slice/crio-152eef17a3f22f68eb64380aa1414873f87aa636e72173280703463843e98214 WatchSource:0}: Error finding container 152eef17a3f22f68eb64380aa1414873f87aa636e72173280703463843e98214: Status 404 returned error can't find the container with id 152eef17a3f22f68eb64380aa1414873f87aa636e72173280703463843e98214
Oct 30 12:44:49.456786 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:49.456764    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbdf052_fa04_4e1c_983a_04b313a0cdfb.slice/crio-02cd5d46eed573ed4f4904ba91544ea05df502d39040ed3ff2df9a21f6043c72 WatchSource:0}: Error finding container 02cd5d46eed573ed4f4904ba91544ea05df502d39040ed3ff2df9a21f6043c72: Status 404 returned error can't find the container with id 02cd5d46eed573ed4f4904ba91544ea05df502d39040ed3ff2df9a21f6043c72
Oct 30 12:44:49.457831 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.457807    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:44:49.462678 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.462659    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:44:49.467393 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:49.467373    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08a5487_7eaa_4438_8fcd_3ead8dc6c20e.slice/crio-02210ac7103cb8fee0c9d6922b3f774b7409cf3c3e38509722954e36eb165f47 WatchSource:0}: Error finding container 02210ac7103cb8fee0c9d6922b3f774b7409cf3c3e38509722954e36eb165f47: Status 404 returned error can't find the container with id 02210ac7103cb8fee0c9d6922b3f774b7409cf3c3e38509722954e36eb165f47
Oct 30 12:44:49.471840 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.471824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:44:49.477149 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.477131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:44:49.492156 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:49.492132    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f5c091_40a2_46e8_9214_ddf02da33d0b.slice/crio-8704082ecaafee1761693042fddc9c101ad08d9f08f9e9af632856b129dfcc58 WatchSource:0}: Error finding container 8704082ecaafee1761693042fddc9c101ad08d9f08f9e9af632856b129dfcc58: Status 404 returned error can't find the container with id 8704082ecaafee1761693042fddc9c101ad08d9f08f9e9af632856b129dfcc58
Oct 30 12:44:49.496474 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:49.496452    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba6439a7_7042_4edf_aa9c_6d33b168873d.slice/crio-953a1a3d1e1476dfed96d1c36d344bdc2d8a7bfdd08e6c3137538bc050a58737 WatchSource:0}: Error finding container 953a1a3d1e1476dfed96d1c36d344bdc2d8a7bfdd08e6c3137538bc050a58737: Status 404 returned error can't find the container with id 953a1a3d1e1476dfed96d1c36d344bdc2d8a7bfdd08e6c3137538bc050a58737
Oct 30 12:44:49.497689 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:49.497655    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccd197b_ec88_445b_8de9_ff83832112bd.slice/crio-2d3d2dc6f41867ef363f9445d7796544e423b55896729f29a7eb077f8bc2f3c7 WatchSource:0}: Error finding container 2d3d2dc6f41867ef363f9445d7796544e423b55896729f29a7eb077f8bc2f3c7: Status 404 returned error can't find the container with id 2d3d2dc6f41867ef363f9445d7796544e423b55896729f29a7eb077f8bc2f3c7
Oct 30 12:44:49.499043 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.499024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:44:49.505317 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.505298    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:44:49.511099 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:49.511070    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429e3a90_1868_4632_8631_5c0207978063.slice/crio-6fdc177b4754e39d1a1dabb5ddb43e599c64550858fca406030492b3a2072321 WatchSource:0}: Error finding container 6fdc177b4754e39d1a1dabb5ddb43e599c64550858fca406030492b3a2072321: Status 404 returned error can't find the container with id 6fdc177b4754e39d1a1dabb5ddb43e599c64550858fca406030492b3a2072321
Oct 30 12:44:49.516510 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.516492    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:44:49.516598 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:44:49.516507    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c3843d_81b7_4554_80ab_f6fc5901f488.slice/crio-12dc6cc351023a89b3b2c74b20fb99e2cf2e49a084ed619bb6247ff9a2fdc8c0 WatchSource:0}: Error finding container 12dc6cc351023a89b3b2c74b20fb99e2cf2e49a084ed619bb6247ff9a2fdc8c0: Status 404 returned error can't find the container with id 12dc6cc351023a89b3b2c74b20fb99e2cf2e49a084ed619bb6247ff9a2fdc8c0
Oct 30 12:44:49.850311 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.849338    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:49.850311 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.849618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:49.850311 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.849780    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:49.850311 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.849836    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret podName:408c5c75-c039-4dc1-89c7-6b0f4412aa59 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:50.849819843 +0000 UTC m=+4.062114353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret") pod "global-pull-secret-syncer-79f2f" (UID: "408c5c75-c039-4dc1-89c7-6b0f4412aa59") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:49.850311 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.850205    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:49.850311 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.850256    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs podName:825a2efb-6a8c-4bdd-bf89-cca7ee710f8e nodeName:}" failed. No retries permitted until 2026-10-30 12:44:50.85024062 +0000 UTC m=+4.062535130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs") pod "network-metrics-daemon-zfp4t" (UID: "825a2efb-6a8c-4bdd-bf89-cca7ee710f8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:49.950606 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:49.950485    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4cv\" (UniqueName: \"kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv\") pod \"network-check-target-7qcht\" (UID: \"34fc63c2-b395-4702-a2e4-134161c0d50c\") " pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:44:49.950753 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.950709    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:44:49.950753 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.950727    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:44:49.950753 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.950739    2576 projected.go:196] Error preparing data for projected volume kube-api-access-gq4cv for pod openshift-network-diagnostics/network-check-target-7qcht: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:49.950899 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:49.950794    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv podName:34fc63c2-b395-4702-a2e4-134161c0d50c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:50.950778132 +0000 UTC m=+4.163072641 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq4cv" (UniqueName: "kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv") pod "network-check-target-7qcht" (UID: "34fc63c2-b395-4702-a2e4-134161c0d50c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:50.180790 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.180727    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:39:49 +0000 UTC" deadline="2028-08-06 01:30:57.947741994 +0000 UTC"
Oct 30 12:44:50.180790 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.180786    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15492h46m7.766960048s"
Oct 30 12:44:50.256936 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.256397    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:50.256936 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:50.256535    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:44:50.268055 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.267860    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp" event={"ID":"f6897566-da42-4b8d-ba85-7ec304624f2b","Type":"ContainerStarted","Data":"ce9500fc0e9de19d8eab6cfa84c7dcdf6aa309e4b785f46c662933584e330b50"}
Oct 30 12:44:50.275364 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.275333    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv" event={"ID":"36c3843d-81b7-4554-80ab-f6fc5901f488","Type":"ContainerStarted","Data":"12dc6cc351023a89b3b2c74b20fb99e2cf2e49a084ed619bb6247ff9a2fdc8c0"}
Oct 30 12:44:50.278834 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.278796    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzgc2" event={"ID":"429e3a90-1868-4632-8631-5c0207978063","Type":"ContainerStarted","Data":"6fdc177b4754e39d1a1dabb5ddb43e599c64550858fca406030492b3a2072321"}
Oct 30 12:44:50.288337 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.288302    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gxdv2" event={"ID":"ba6439a7-7042-4edf-aa9c-6d33b168873d","Type":"ContainerStarted","Data":"953a1a3d1e1476dfed96d1c36d344bdc2d8a7bfdd08e6c3137538bc050a58737"}
Oct 30 12:44:50.295771 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.295715    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fd9xb" event={"ID":"d08a5487-7eaa-4438-8fcd-3ead8dc6c20e","Type":"ContainerStarted","Data":"02210ac7103cb8fee0c9d6922b3f774b7409cf3c3e38509722954e36eb165f47"}
Oct 30 12:44:50.313886 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.313841    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x48w4" event={"ID":"9bbdf052-fa04-4e1c-983a-04b313a0cdfb","Type":"ContainerStarted","Data":"02cd5d46eed573ed4f4904ba91544ea05df502d39040ed3ff2df9a21f6043c72"}
Oct 30 12:44:50.330843 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.330762    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jtfpn" event={"ID":"a8053a33-e1bf-43f4-b23b-34694e7ce042","Type":"ContainerStarted","Data":"152eef17a3f22f68eb64380aa1414873f87aa636e72173280703463843e98214"}
Oct 30 12:44:50.343727 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.343690    2576 generic.go:356] "Generic (PLEG): container finished" podID="e8bd9c92b47bd47faf7dc12d1bdd0a47" containerID="4aeda4a396f0aee7a6eb95eea247d68b20c3fadbfda23e45bf5d714dfc1def15" exitCode=0
Oct 30 12:44:50.343880 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.343786    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal" event={"ID":"e8bd9c92b47bd47faf7dc12d1bdd0a47","Type":"ContainerDied","Data":"4aeda4a396f0aee7a6eb95eea247d68b20c3fadbfda23e45bf5d714dfc1def15"}
Oct 30 12:44:50.352133 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.351657    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal" event={"ID":"6886c36025c26ee79576270d5200a014","Type":"ContainerStarted","Data":"09c06cb8f6ffe92d449bfd7a645ef115caecaf57fbfafd62f913842100bd71d5"}
Oct 30 12:44:50.354501 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.354456    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bzkw4" event={"ID":"dccd197b-ec88-445b-8de9-ff83832112bd","Type":"ContainerStarted","Data":"2d3d2dc6f41867ef363f9445d7796544e423b55896729f29a7eb077f8bc2f3c7"}
Oct 30 12:44:50.363443 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.363366    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tg5ns" event={"ID":"52f5c091-40a2-46e8-9214-ddf02da33d0b","Type":"ContainerStarted","Data":"8704082ecaafee1761693042fddc9c101ad08d9f08f9e9af632856b129dfcc58"}
Oct 30 12:44:50.856349 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.856047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:50.856349 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.856139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:50.856349 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:50.856266    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:50.856349 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:50.856353    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs podName:825a2efb-6a8c-4bdd-bf89-cca7ee710f8e nodeName:}" failed. No retries permitted until 2026-10-30 12:44:52.856334374 +0000 UTC m=+6.068628896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs") pod "network-metrics-daemon-zfp4t" (UID: "825a2efb-6a8c-4bdd-bf89-cca7ee710f8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:50.856576 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:50.856419    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:50.856576 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:50.856449    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret podName:408c5c75-c039-4dc1-89c7-6b0f4412aa59 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:52.856439515 +0000 UTC m=+6.068734037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret") pod "global-pull-secret-syncer-79f2f" (UID: "408c5c75-c039-4dc1-89c7-6b0f4412aa59") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:50.956697 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:50.956660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4cv\" (UniqueName: \"kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv\") pod \"network-check-target-7qcht\" (UID: \"34fc63c2-b395-4702-a2e4-134161c0d50c\") " pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:44:50.956889 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:50.956872    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:44:50.956956 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:50.956896    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:44:50.956956 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:50.956909    2576 projected.go:196] Error preparing data for projected volume kube-api-access-gq4cv for pod openshift-network-diagnostics/network-check-target-7qcht: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:50.957053 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:50.956964    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv podName:34fc63c2-b395-4702-a2e4-134161c0d50c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:52.95694722 +0000 UTC m=+6.169241734 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq4cv" (UniqueName: "kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv") pod "network-check-target-7qcht" (UID: "34fc63c2-b395-4702-a2e4-134161c0d50c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:51.181217 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:51.181128    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:39:49 +0000 UTC" deadline="2028-07-21 01:11:24.09907537 +0000 UTC"
Oct 30 12:44:51.181217 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:51.181171    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15108h26m32.917909052s"
Oct 30 12:44:51.256547 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:51.256517    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:44:51.256680 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:51.256657    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:44:51.257529 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:51.257390    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:51.257529 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:51.257492    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:44:51.376391 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:51.376328    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal" event={"ID":"e8bd9c92b47bd47faf7dc12d1bdd0a47","Type":"ContainerStarted","Data":"8b19358587860669d595e4c041a6d3d7f512d54adff3d426e48bf7972bd9bc43"}
Oct 30 12:44:51.413401 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:51.413195    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal" podStartSLOduration=3.4131739039999998 podStartE2EDuration="3.413173904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:50.417614807 +0000 UTC m=+3.629909338" watchObservedRunningTime="2026-10-30 12:44:51.413173904 +0000 UTC m=+4.625468433"
Oct 30 12:44:51.414098 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:51.414055    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal" podStartSLOduration=3.414044636 podStartE2EDuration="3.414044636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:51.413337412 +0000 UTC m=+4.625631969" watchObservedRunningTime="2026-10-30 12:44:51.414044636 +0000 UTC m=+4.626339170"
Oct 30 12:44:52.257381 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:52.256041    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:52.257381 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:52.256193    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:44:52.382421 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:52.381605    2576 generic.go:356] "Generic (PLEG): container finished" podID="429e3a90-1868-4632-8631-5c0207978063" containerID="f2ddbdc2aee542040e3b54d20e69e2344a35139c6067622d326fda789888ae0b" exitCode=0
Oct 30 12:44:52.390679 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:52.390412    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzgc2" event={"ID":"429e3a90-1868-4632-8631-5c0207978063","Type":"ContainerDied","Data":"f2ddbdc2aee542040e3b54d20e69e2344a35139c6067622d326fda789888ae0b"}
Oct 30 12:44:52.876726 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:52.875782    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:52.876726 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:52.875875    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:52.876726 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:52.875995    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:52.876726 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:52.876062    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs podName:825a2efb-6a8c-4bdd-bf89-cca7ee710f8e nodeName:}" failed. No retries permitted until 2026-10-30 12:44:56.876042909 +0000 UTC m=+10.088337418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs") pod "network-metrics-daemon-zfp4t" (UID: "825a2efb-6a8c-4bdd-bf89-cca7ee710f8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:52.876726 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:52.876575    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:52.876726 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:52.876648    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret podName:408c5c75-c039-4dc1-89c7-6b0f4412aa59 nodeName:}" failed. No retries permitted until 2026-10-30 12:44:56.876618147 +0000 UTC m=+10.088912656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret") pod "global-pull-secret-syncer-79f2f" (UID: "408c5c75-c039-4dc1-89c7-6b0f4412aa59") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:52.976908 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:52.976846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4cv\" (UniqueName: \"kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv\") pod \"network-check-target-7qcht\" (UID: \"34fc63c2-b395-4702-a2e4-134161c0d50c\") " pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:44:52.977076 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:52.977007    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:44:52.977076 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:52.977026    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:44:52.977076 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:52.977038    2576 projected.go:196] Error preparing data for projected volume kube-api-access-gq4cv for pod openshift-network-diagnostics/network-check-target-7qcht: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:52.977292 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:52.977094    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv podName:34fc63c2-b395-4702-a2e4-134161c0d50c nodeName:}" failed. No retries permitted until 2026-10-30 12:44:56.977077701 +0000 UTC m=+10.189372216 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq4cv" (UniqueName: "kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv") pod "network-check-target-7qcht" (UID: "34fc63c2-b395-4702-a2e4-134161c0d50c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:53.258333 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:53.258302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:44:53.258869 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:53.258427    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:44:53.258869 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:53.258787    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:53.258984 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:53.258877    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:44:53.390152 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:53.390066    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fd9xb" event={"ID":"d08a5487-7eaa-4438-8fcd-3ead8dc6c20e","Type":"ContainerStarted","Data":"e5cf02c2ad83edd5e639a71c50b5e37b280a72e0a7ff580841e73164851d54ab"}
Oct 30 12:44:53.427550 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:53.427498    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fd9xb" podStartSLOduration=2.932101194 podStartE2EDuration="6.427483744s" totalImagesPullingTime="3.49538255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:53.427238877 +0000 UTC m=+6.639533411" watchObservedRunningTime="2026-10-30 12:44:53.427483744 +0000 UTC m=+6.639778273"
Oct 30 12:44:54.256313 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:54.256253    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:54.256466 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:54.256426    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:44:55.258547 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:55.258512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:44:55.258897 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:55.258644    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:44:55.258950 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:55.258940    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:55.259021 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:55.259005    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:44:55.405730 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:55.404973    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bzkw4" event={"ID":"dccd197b-ec88-445b-8de9-ff83832112bd","Type":"ContainerStarted","Data":"385b7ffc383c9ce5e634d3b51df1ecad12067eb03ef52e6620668d99531eceee"}
Oct 30 12:44:55.440795 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:55.440745    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bzkw4" podStartSLOduration=3.719949991 podStartE2EDuration="8.440731492s" totalImagesPullingTime="4.720781501s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:55.440525089 +0000 UTC m=+8.652819621" watchObservedRunningTime="2026-10-30 12:44:55.440731492 +0000 UTC m=+8.653026076"
Oct 30 12:44:56.256427 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:56.256350    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:56.256591 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:56.256520    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:44:56.414903 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:56.414769    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv" event={"ID":"36c3843d-81b7-4554-80ab-f6fc5901f488","Type":"ContainerStarted","Data":"a439b9436270290718a8e840216049c0e49b0d08fe23a763e214bb7c1510556a"}
Oct 30 12:44:56.893596 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:56.893528    2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 12:44:56.912701 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:56.912244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:56.912701 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:56.912375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:56.912701 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:56.912501    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:56.912701 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:56.912564    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs podName:825a2efb-6a8c-4bdd-bf89-cca7ee710f8e nodeName:}" failed. No retries permitted until 2026-10-30 12:45:04.912544509 +0000 UTC m=+18.124839038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs") pod "network-metrics-daemon-zfp4t" (UID: "825a2efb-6a8c-4bdd-bf89-cca7ee710f8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:44:56.912701 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:56.912628    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:56.912701 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:56.912659    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret podName:408c5c75-c039-4dc1-89c7-6b0f4412aa59 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:04.912648257 +0000 UTC m=+18.124942780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret") pod "global-pull-secret-syncer-79f2f" (UID: "408c5c75-c039-4dc1-89c7-6b0f4412aa59") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:44:57.013496 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:57.013408    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4cv\" (UniqueName: \"kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv\") pod \"network-check-target-7qcht\" (UID: \"34fc63c2-b395-4702-a2e4-134161c0d50c\") " pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:44:57.013645 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:57.013621    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:44:57.013715 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:57.013648    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:44:57.013715 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:57.013662    2576 projected.go:196] Error preparing data for projected volume kube-api-access-gq4cv for pod openshift-network-diagnostics/network-check-target-7qcht: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:57.013805 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:57.013723    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv podName:34fc63c2-b395-4702-a2e4-134161c0d50c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:05.013705487 +0000 UTC m=+18.226000001 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq4cv" (UniqueName: "kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv") pod "network-check-target-7qcht" (UID: "34fc63c2-b395-4702-a2e4-134161c0d50c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:44:57.067568 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:57.067540    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:44:57.068367 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:57.068344    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:44:57.178638 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:57.177946    2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:44:56.893552658Z","UUID":"2b123e59-60d6-4829-b36b-c8979156fd07","Handler":null,"Name":"","Endpoint":""}
Oct 30 12:44:57.182429 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:57.182407    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 12:44:57.182429 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:57.182434    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 12:44:57.259335 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:57.258364    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:57.259335 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:57.258699    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:44:57.259335 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:57.258526    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:44:57.259763 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:57.259703    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:44:57.421192 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:57.421104    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv" event={"ID":"36c3843d-81b7-4554-80ab-f6fc5901f488","Type":"ContainerStarted","Data":"998ab449e95a03c395debc53ba1acac4a898eaf552a1462441c18d0c080dea56"}
Oct 30 12:44:57.424998 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:57.424045    2576 generic.go:356] "Generic (PLEG): container finished" podID="429e3a90-1868-4632-8631-5c0207978063" containerID="02fd14a51f8d2c34d09b1a0ca6389849547c39625ef73fb76a00bb6955d1e7ca" exitCode=0
Oct 30 12:44:57.424998 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:57.424124    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzgc2" event={"ID":"429e3a90-1868-4632-8631-5c0207978063","Type":"ContainerDied","Data":"02fd14a51f8d2c34d09b1a0ca6389849547c39625ef73fb76a00bb6955d1e7ca"}
Oct 30 12:44:57.429258 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:57.429204    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-x48w4" event={"ID":"9bbdf052-fa04-4e1c-983a-04b313a0cdfb","Type":"ContainerStarted","Data":"fc1c595880b39408307da7f5a051985360600f04b10ea0958bb4521803f7c155"}
Oct 30 12:44:58.196090 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:58.196063    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:44:58.196815 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:58.196796    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:44:58.234801 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:58.234743    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-x48w4" podStartSLOduration=3.902240124 podStartE2EDuration="11.234724349s" totalImagesPullingTime="7.332484225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:57.518635203 +0000 UTC m=+10.730929732" watchObservedRunningTime="2026-10-30 12:44:58.234724349 +0000 UTC m=+11.447018878"
Oct 30 12:44:58.256586 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:58.256556    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:44:58.256760 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:58.256674    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:44:58.433353 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:58.432858    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tg5ns" event={"ID":"52f5c091-40a2-46e8-9214-ddf02da33d0b","Type":"ContainerStarted","Data":"8d8e429bca2579c2f6860479304c2028810652f59dcf9495170d705c8f4e57d2"}
Oct 30 12:44:58.435478 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:58.435448    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv" event={"ID":"36c3843d-81b7-4554-80ab-f6fc5901f488","Type":"ContainerStarted","Data":"52b2d6e1264ee9dac7fe40f816d4226aea36e56ae905437569767d65c4317e25"}
Oct 30 12:44:58.438105 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:58.438078    2576 generic.go:356] "Generic (PLEG): container finished" podID="429e3a90-1868-4632-8631-5c0207978063" containerID="dadf45da29b49d09436574031b6626e88ffafe3094539a475029821fee8ce8c1" exitCode=0
Oct 30 12:44:58.438192 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:58.438161    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzgc2" event={"ID":"429e3a90-1868-4632-8631-5c0207978063","Type":"ContainerDied","Data":"dadf45da29b49d09436574031b6626e88ffafe3094539a475029821fee8ce8c1"}
Oct 30 12:44:58.472561 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:58.472510    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-tg5ns" podStartSLOduration=3.560238504 podStartE2EDuration="11.47249612s" totalImagesPullingTime="7.912257616s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:58.471930281 +0000 UTC m=+11.684224878" watchObservedRunningTime="2026-10-30 12:44:58.47249612 +0000 UTC m=+11.684790649"
Oct 30 12:44:59.256389 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:59.256309    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:44:59.256389 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:59.256356    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:44:59.256570 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:59.256430    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:44:59.256619 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:44:59.256574    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:44:59.444236 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:59.444199    2576 generic.go:356] "Generic (PLEG): container finished" podID="429e3a90-1868-4632-8631-5c0207978063" containerID="91f48fe4a7a3c66734b6627edbcbcf817cd532eaa58a614c4cf82ab80d758597" exitCode=0
Oct 30 12:44:59.444920 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:59.444290    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzgc2" event={"ID":"429e3a90-1868-4632-8631-5c0207978063","Type":"ContainerDied","Data":"91f48fe4a7a3c66734b6627edbcbcf817cd532eaa58a614c4cf82ab80d758597"}
Oct 30 12:44:59.446185 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:59.446154    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gxdv2" event={"ID":"ba6439a7-7042-4edf-aa9c-6d33b168873d","Type":"ContainerStarted","Data":"0dbf046c36df24304503eab06b653d9b02275c6742f3de0b723b95b5c32a9517"}
Oct 30 12:44:59.447858 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:59.447835    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-jtfpn" event={"ID":"a8053a33-e1bf-43f4-b23b-34694e7ce042","Type":"ContainerStarted","Data":"56d889efc9bdf785345a17769c5625464825a4c1f8f0d6813fd17f86469140d7"}
Oct 30 12:44:59.491839 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:59.491787    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv" podStartSLOduration=4.595585379 podStartE2EDuration="12.491767931s" totalImagesPullingTime="7.896182552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:58.558440519 +0000 UTC m=+11.770735049" watchObservedRunningTime="2026-10-30 12:44:59.491767931 +0000 UTC m=+12.704062462"
Oct 30 12:44:59.527843 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:59.527751    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-jtfpn" podStartSLOduration=4.773293441 podStartE2EDuration="12.527738009s" totalImagesPullingTime="7.754444568s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:59.52733562 +0000 UTC m=+12.739630151" watchObservedRunningTime="2026-10-30 12:44:59.527738009 +0000 UTC m=+12.740032539"
Oct 30 12:44:59.566014 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:44:59.565943    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gxdv2" podStartSLOduration=2.980996064 podStartE2EDuration="12.565926109s" totalImagesPullingTime="9.584930045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:44:59.565558143 +0000 UTC m=+12.777852672" watchObservedRunningTime="2026-10-30 12:44:59.565926109 +0000 UTC m=+12.778220660"
Oct 30 12:45:00.256753 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:00.256721    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:00.256923 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:00.256842    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:01.256330 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:01.255833    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:01.256330 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:01.255852    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:01.256330 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:01.255974    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:45:01.256330 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:01.256198    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:45:01.455586 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:01.455293    2576 generic.go:356] "Generic (PLEG): container finished" podID="429e3a90-1868-4632-8631-5c0207978063" containerID="48a7c7cae6da64cdba507f18d16fd403026286637b4cda7b1555829d695ad085" exitCode=0
Oct 30 12:45:01.455586 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:01.455478    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzgc2" event={"ID":"429e3a90-1868-4632-8631-5c0207978063","Type":"ContainerDied","Data":"48a7c7cae6da64cdba507f18d16fd403026286637b4cda7b1555829d695ad085"}
Oct 30 12:45:02.256660 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:02.256628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:02.257040 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:02.256766    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:02.461718 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:02.461677    2576 generic.go:356] "Generic (PLEG): container finished" podID="429e3a90-1868-4632-8631-5c0207978063" containerID="01e224997aa4ad2c8ccf08ac6523e3c76f4d73087390a72f2ce9eace4539f18b" exitCode=0
Oct 30 12:45:02.461879 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:02.461731    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzgc2" event={"ID":"429e3a90-1868-4632-8631-5c0207978063","Type":"ContainerDied","Data":"01e224997aa4ad2c8ccf08ac6523e3c76f4d73087390a72f2ce9eace4539f18b"}
Oct 30 12:45:03.255986 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:03.255954    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:03.256185 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:03.256083    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:45:03.256185 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:03.256136    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:03.256321 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:03.256249    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:45:03.469527 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:03.469494    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jzgc2" event={"ID":"429e3a90-1868-4632-8631-5c0207978063","Type":"ContainerStarted","Data":"84497d3c4656aa94c30b24ec6524b9e3e9e49d04d610cfe0f7acbbc1e97fe29b"}
Oct 30 12:45:03.543985 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:03.543839    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jzgc2" podStartSLOduration=6.552254654 podStartE2EDuration="16.543823021s" totalImagesPullingTime="9.991568367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:03.543468232 +0000 UTC m=+16.755762763" watchObservedRunningTime="2026-10-30 12:45:03.543823021 +0000 UTC m=+16.756117552"
Oct 30 12:45:04.033495 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:04.033332    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tg5ns_52f5c091-40a2-46e8-9214-ddf02da33d0b/dns-node-resolver/0.log"
Oct 30 12:45:04.256644 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:04.256577    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:04.256800 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:04.256681    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:04.474651 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:04.474623    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 12:45:04.475058 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:04.474952    2576 generic.go:356] "Generic (PLEG): container finished" podID="f6897566-da42-4b8d-ba85-7ec304624f2b" containerID="b9a72fd6984e27eaa724167e0ae42f05f2f36717495836ab9e6c2409c87d4656" exitCode=1
Oct 30 12:45:04.475058 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:04.475026    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp" event={"ID":"f6897566-da42-4b8d-ba85-7ec304624f2b","Type":"ContainerStarted","Data":"4872030c20976843803342596a401894bf8b117352faf304d10b00836d4b670f"}
Oct 30 12:45:04.475131 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:04.475088    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp" event={"ID":"f6897566-da42-4b8d-ba85-7ec304624f2b","Type":"ContainerStarted","Data":"7399a43fdb945bb980138fc9b231fc51cf9bf99730e39f01dff8075a0bbf8723"}
Oct 30 12:45:04.475131 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:04.475118    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp" event={"ID":"f6897566-da42-4b8d-ba85-7ec304624f2b","Type":"ContainerStarted","Data":"dd7221ffcb310e1cff1b81c179940a751bdb104b771e37eebbd7b79bdd342b94"}
Oct 30 12:45:04.475196 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:04.475149    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp" event={"ID":"f6897566-da42-4b8d-ba85-7ec304624f2b","Type":"ContainerStarted","Data":"a05a39f1a89cdb0c99dc1a36f16852e1379156488cbb47ed215f643e0b877982"}
Oct 30 12:45:04.475196 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:04.475175    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp" event={"ID":"f6897566-da42-4b8d-ba85-7ec304624f2b","Type":"ContainerDied","Data":"b9a72fd6984e27eaa724167e0ae42f05f2f36717495836ab9e6c2409c87d4656"}
Oct 30 12:45:04.475265 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:04.475204    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp" event={"ID":"f6897566-da42-4b8d-ba85-7ec304624f2b","Type":"ContainerStarted","Data":"0adc00708ada0b2448bc816dcc1491b1979f6f468e0ea482ab97373f24a83b12"}
Oct 30 12:45:04.772598 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:04.772573    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bzkw4_dccd197b-ec88-445b-8de9-ff83832112bd/node-ca/0.log"
Oct 30 12:45:04.978198 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:04.978170    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:04.978354 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:04.978214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:04.978354 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:04.978339    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:45:04.978354 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:04.978339    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:45:04.978475 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:04.978405    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret podName:408c5c75-c039-4dc1-89c7-6b0f4412aa59 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:20.978388031 +0000 UTC m=+34.190682553 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret") pod "global-pull-secret-syncer-79f2f" (UID: "408c5c75-c039-4dc1-89c7-6b0f4412aa59") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:45:04.978475 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:04.978420    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs podName:825a2efb-6a8c-4bdd-bf89-cca7ee710f8e nodeName:}" failed. No retries permitted until 2026-10-30 12:45:20.97841311 +0000 UTC m=+34.190707618 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs") pod "network-metrics-daemon-zfp4t" (UID: "825a2efb-6a8c-4bdd-bf89-cca7ee710f8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:45:05.079393 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:05.079317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4cv\" (UniqueName: \"kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv\") pod \"network-check-target-7qcht\" (UID: \"34fc63c2-b395-4702-a2e4-134161c0d50c\") " pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:05.079525 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:05.079468    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:45:05.079525 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:05.079489    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:45:05.079525 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:05.079498    2576 projected.go:196] Error preparing data for projected volume kube-api-access-gq4cv for pod openshift-network-diagnostics/network-check-target-7qcht: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:45:05.079635 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:05.079546    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv podName:34fc63c2-b395-4702-a2e4-134161c0d50c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:21.079533202 +0000 UTC m=+34.291827711 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq4cv" (UniqueName: "kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv") pod "network-check-target-7qcht" (UID: "34fc63c2-b395-4702-a2e4-134161c0d50c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:45:05.256061 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:05.256030    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:05.256061 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:05.256067    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:05.256389 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:05.256160    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:45:05.256389 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:05.256329    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:45:05.257097 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:05.257074    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fd9xb_d08a5487-7eaa-4438-8fcd-3ead8dc6c20e/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:05.397228 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:05.397159    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fd9xb_d08a5487-7eaa-4438-8fcd-3ead8dc6c20e/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:05.586533 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:05.586505    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fd9xb_d08a5487-7eaa-4438-8fcd-3ead8dc6c20e/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:05.988258 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:05.988224    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fd9xb_d08a5487-7eaa-4438-8fcd-3ead8dc6c20e/konnectivity-agent/0.log" limit=1024
Oct 30 12:45:06.256150 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:06.256075    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:06.256299 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:06.256182    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:06.482188 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:06.482162    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 12:45:06.482526 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:06.482506    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp" event={"ID":"f6897566-da42-4b8d-ba85-7ec304624f2b","Type":"ContainerStarted","Data":"08b565696f0ee578c30c19f5a02c789c94410c192f76003fc7a45119000c8af3"}
Oct 30 12:45:07.257079 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:07.257043    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:07.257524 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:07.257155    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:45:07.257524 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:07.257204    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:07.257524 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:07.257304    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:45:08.255686 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:08.255658    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:08.255832 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:08.255771    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:08.490328 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:08.490249    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 12:45:08.490901 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:08.490616    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp" event={"ID":"f6897566-da42-4b8d-ba85-7ec304624f2b","Type":"ContainerStarted","Data":"dd61258715b07ec0daf221661e010d100ef14ecc3232d56381f160e18a81374c"}
Oct 30 12:45:08.490901 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:08.490871    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:08.491014 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:08.490925    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:08.491061 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:08.491043    2576 scope.go:122] "RemoveContainer" containerID="b9a72fd6984e27eaa724167e0ae42f05f2f36717495836ab9e6c2409c87d4656"
Oct 30 12:45:08.507928 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:08.507907    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:09.255857 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:09.255821    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:09.256027 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:09.255927    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:45:09.256027 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:09.255978    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:09.256129 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:09.256061    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:45:09.496439 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:09.496412    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 12:45:09.496833 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:09.496747    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp" event={"ID":"f6897566-da42-4b8d-ba85-7ec304624f2b","Type":"ContainerStarted","Data":"c0ce02634998867de7bce41da88cc335c38c86e8f7273a716e808e61f564615d"}
Oct 30 12:45:09.496978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:09.496953    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:09.512413 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:09.512357    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:09.556586 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:09.556540    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp" podStartSLOduration=8.481168269 podStartE2EDuration="22.556526775s" totalImagesPullingTime="14.075358506s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:09.556159007 +0000 UTC m=+22.768453536" watchObservedRunningTime="2026-10-30 12:45:09.556526775 +0000 UTC m=+22.768821304"
Oct 30 12:45:10.256452 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:10.256417    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:10.256649 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:10.256534    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:11.256418 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:11.256383    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:11.256871 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:11.256514    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:45:11.256871 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:11.256560    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:11.256871 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:11.256661    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:45:12.256525 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:12.256492    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:12.256879 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:12.256598    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:13.255993 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:13.255958    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:13.256227 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:13.256001    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:13.256227 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:13.256089    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:45:13.256227 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:13.256187    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:45:14.255889 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:14.255860    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:14.256244 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:14.255958    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:15.256522 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:15.256488    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:15.257126 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:15.256491    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:15.257126 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:15.256588    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:45:15.257126 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:15.256645    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:45:16.256762 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:16.256729    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:16.257108 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:16.256839    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:17.257443 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:17.257413    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:17.257812 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:17.257509    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:45:17.257812 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:17.257596    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:17.257812 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:17.257677    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:45:18.255893 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:18.255862    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:18.256048 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:18.255972    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:19.258769 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:19.258730    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:19.259234 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:19.258848    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:45:19.259234 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:19.258892    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:19.259234 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:19.258973    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:45:20.256014 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:20.255979    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:20.256178 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:20.256093    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:20.990884 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:20.990852    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:20.991344 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:20.990908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:20.991344 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:20.990995    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:45:20.991344 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:20.990997    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 12:45:20.991344 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:20.991046    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs podName:825a2efb-6a8c-4bdd-bf89-cca7ee710f8e nodeName:}" failed. No retries permitted until 2026-10-30 12:45:52.991032567 +0000 UTC m=+66.203327089 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs") pod "network-metrics-daemon-zfp4t" (UID: "825a2efb-6a8c-4bdd-bf89-cca7ee710f8e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 12:45:20.991344 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:20.991059    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret podName:408c5c75-c039-4dc1-89c7-6b0f4412aa59 nodeName:}" failed. No retries permitted until 2026-10-30 12:45:52.991053571 +0000 UTC m=+66.203348079 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret") pod "global-pull-secret-syncer-79f2f" (UID: "408c5c75-c039-4dc1-89c7-6b0f4412aa59") : object "kube-system"/"original-pull-secret" not registered
Oct 30 12:45:21.091768 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:21.091732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4cv\" (UniqueName: \"kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv\") pod \"network-check-target-7qcht\" (UID: \"34fc63c2-b395-4702-a2e4-134161c0d50c\") " pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:21.091936 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:21.091886    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 12:45:21.091936 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:21.091903    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 12:45:21.091936 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:21.091912    2576 projected.go:196] Error preparing data for projected volume kube-api-access-gq4cv for pod openshift-network-diagnostics/network-check-target-7qcht: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:45:21.092035 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:21.091962    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv podName:34fc63c2-b395-4702-a2e4-134161c0d50c nodeName:}" failed. No retries permitted until 2026-10-30 12:45:53.091949467 +0000 UTC m=+66.304243994 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gq4cv" (UniqueName: "kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv") pod "network-check-target-7qcht" (UID: "34fc63c2-b395-4702-a2e4-134161c0d50c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 12:45:21.258749 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:21.258678    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:21.258882 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:21.258679    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:21.258882 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:21.258771    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:45:21.258882 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:21.258838    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:45:22.255884 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:22.255823    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:22.256234 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:22.255948    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:22.317446 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:22.317418    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-79f2f"]
Oct 30 12:45:22.317621 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:22.317523    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:22.317621 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:22.317607    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:45:22.323465 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:22.323432    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zfp4t"]
Oct 30 12:45:22.323580 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:22.323547    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7qcht"]
Oct 30 12:45:22.323668 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:22.323652    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:22.323824 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:22.323798    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:45:22.523077 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:22.522995    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:22.523212 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:22.523113    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:24.256798 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:24.256763    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:24.256798 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:24.256794    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:24.257237 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:24.256873    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:24.257237 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:24.256912    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:24.257237 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:24.256978    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:45:24.257237 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:24.257037    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:45:26.256762 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:26.256587    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:26.257205 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:26.256587    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:26.257205 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:26.256839    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7qcht" podUID="34fc63c2-b395-4702-a2e4-134161c0d50c"
Oct 30 12:45:26.257205 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:26.256589    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:26.257205 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:26.256910    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zfp4t" podUID="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e"
Oct 30 12:45:26.257205 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:26.257017    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-79f2f" podUID="408c5c75-c039-4dc1-89c7-6b0f4412aa59"
Oct 30 12:45:27.549401 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.549370    2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeReady"
Oct 30 12:45:27.549870 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.549514    2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 12:45:27.614226 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.614137    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rm7hq"]
Oct 30 12:45:27.636282 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.636257    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k47jk"]
Oct 30 12:45:27.636497 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.636397    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:27.650390 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.650361    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rm7hq"]
Oct 30 12:45:27.650495 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.650417    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k47jk"]
Oct 30 12:45:27.650495 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.650486    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.650590 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.650498    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 12:45:27.650590 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.650501    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mxwzk\""
Oct 30 12:45:27.650765 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.650747    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 12:45:27.669089 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.669060    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 12:45:27.669309 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.669289    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 12:45:27.669439 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.669420    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g92fm\""
Oct 30 12:45:27.673163 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.673127    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 12:45:27.673163 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.673127    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 12:45:27.713517 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.713490    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-dlv29"]
Oct 30 12:45:27.726037 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.726014    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dlv29"
Oct 30 12:45:27.734460 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.734438    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-data-volume\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.734563 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.734465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.734563 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.734489    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-crio-socket\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.734563 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.734505    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8c3435-b6de-4aea-b596-970c9e17985c-config-volume\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:27.734563 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.734526    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c8c3435-b6de-4aea-b596-970c9e17985c-metrics-tls\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:27.734689 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.734651    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.734720 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.734693    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vkrz\" (UniqueName: \"kubernetes.io/projected/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-kube-api-access-6vkrz\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.734748 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.734739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c8c3435-b6de-4aea-b596-970c9e17985c-tmp-dir\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:27.734844 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.734831    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkwhq\" (UniqueName: \"kubernetes.io/projected/6c8c3435-b6de-4aea-b596-970c9e17985c-kube-api-access-rkwhq\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:27.736197 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.736178    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dlv29"]
Oct 30 12:45:27.750681 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.750654    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 12:45:27.750815 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.750795    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 12:45:27.750867 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.750798    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 12:45:27.750867 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.750845    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xlfj2\""
Oct 30 12:45:27.835583 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.835503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.835583 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.835534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkrz\" (UniqueName: \"kubernetes.io/projected/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-kube-api-access-6vkrz\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.835583 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.835574    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sllfq\" (UniqueName: \"kubernetes.io/projected/b0eab2da-9812-424c-9be0-af423308d43c-kube-api-access-sllfq\") pod \"ingress-canary-dlv29\" (UID: \"b0eab2da-9812-424c-9be0-af423308d43c\") " pod="openshift-ingress-canary/ingress-canary-dlv29"
Oct 30 12:45:27.835794 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.835607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0eab2da-9812-424c-9be0-af423308d43c-cert\") pod \"ingress-canary-dlv29\" (UID: \"b0eab2da-9812-424c-9be0-af423308d43c\") " pod="openshift-ingress-canary/ingress-canary-dlv29"
Oct 30 12:45:27.835794 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.835637    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c8c3435-b6de-4aea-b596-970c9e17985c-tmp-dir\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:27.835794 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.835727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkwhq\" (UniqueName: \"kubernetes.io/projected/6c8c3435-b6de-4aea-b596-970c9e17985c-kube-api-access-rkwhq\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:27.835908 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.835794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-data-volume\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.835908 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.835823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.835908 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.835864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-crio-socket\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.835908 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.835890    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8c3435-b6de-4aea-b596-970c9e17985c-config-volume\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:27.836073 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.835911    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c8c3435-b6de-4aea-b596-970c9e17985c-metrics-tls\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:27.836073 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.835973    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c8c3435-b6de-4aea-b596-970c9e17985c-tmp-dir\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:27.836157 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.836099    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-crio-socket\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.836239 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.836214    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-data-volume\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.836459 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.836437    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.836529 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.836457    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8c3435-b6de-4aea-b596-970c9e17985c-config-volume\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:27.839620 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.839601    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.839707 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.839601    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c8c3435-b6de-4aea-b596-970c9e17985c-metrics-tls\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:27.856582 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.856555    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkrz\" (UniqueName: \"kubernetes.io/projected/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-kube-api-access-6vkrz\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:27.856767 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.856748    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkwhq\" (UniqueName: \"kubernetes.io/projected/6c8c3435-b6de-4aea-b596-970c9e17985c-kube-api-access-rkwhq\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:27.936909 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.936875    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0eab2da-9812-424c-9be0-af423308d43c-cert\") pod \"ingress-canary-dlv29\" (UID: \"b0eab2da-9812-424c-9be0-af423308d43c\") " pod="openshift-ingress-canary/ingress-canary-dlv29"
Oct 30 12:45:27.937052 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.936983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sllfq\" (UniqueName: \"kubernetes.io/projected/b0eab2da-9812-424c-9be0-af423308d43c-kube-api-access-sllfq\") pod \"ingress-canary-dlv29\" (UID: \"b0eab2da-9812-424c-9be0-af423308d43c\") " pod="openshift-ingress-canary/ingress-canary-dlv29"
Oct 30 12:45:27.939187 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.939168    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0eab2da-9812-424c-9be0-af423308d43c-cert\") pod \"ingress-canary-dlv29\" (UID: \"b0eab2da-9812-424c-9be0-af423308d43c\") " pod="openshift-ingress-canary/ingress-canary-dlv29"
Oct 30 12:45:27.958607 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.958582    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sllfq\" (UniqueName: \"kubernetes.io/projected/b0eab2da-9812-424c-9be0-af423308d43c-kube-api-access-sllfq\") pod \"ingress-canary-dlv29\" (UID: \"b0eab2da-9812-424c-9be0-af423308d43c\") " pod="openshift-ingress-canary/ingress-canary-dlv29"
Oct 30 12:45:27.971583 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:27.971564    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:28.023003 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.022973    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:28.045081 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.045055    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-dlv29"
Oct 30 12:45:28.256462 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.256435    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:28.256462 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.256459    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:28.256658 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.256468    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:28.269938 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.269914    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zngfb\""
Oct 30 12:45:28.270043 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.269914    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:45:28.270043 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.269941    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:45:28.270043 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.269954    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:45:28.270043 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.269966    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:45:28.270043 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.269954    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w75sr\""
Oct 30 12:45:28.270043 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.269961    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5fhhc\""
Oct 30 12:45:28.355436 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.355367    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rm7hq"]
Oct 30 12:45:28.358081 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.357814    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k47jk"]
Oct 30 12:45:28.361336 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:45:28.361314    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8c3435_b6de_4aea_b596_970c9e17985c.slice/crio-b3c29c3db2ac1fc0c5bda0cdd5d028040453bfd2da80822c25c6946f7cf4f3bd WatchSource:0}: Error finding container b3c29c3db2ac1fc0c5bda0cdd5d028040453bfd2da80822c25c6946f7cf4f3bd: Status 404 returned error can't find the container with id b3c29c3db2ac1fc0c5bda0cdd5d028040453bfd2da80822c25c6946f7cf4f3bd
Oct 30 12:45:28.362960 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:45:28.362908    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb5dfe9c_5cf0_43af_875c_3bc0fd07c265.slice/crio-4eab01a9fa445d90b530a1373c3ffa1610fa98e01cba407c987b00827a6edc17 WatchSource:0}: Error finding container 4eab01a9fa445d90b530a1373c3ffa1610fa98e01cba407c987b00827a6edc17: Status 404 returned error can't find the container with id 4eab01a9fa445d90b530a1373c3ffa1610fa98e01cba407c987b00827a6edc17
Oct 30 12:45:28.370736 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.370714    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-dlv29"]
Oct 30 12:45:28.375326 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:45:28.375162    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0eab2da_9812_424c_9be0_af423308d43c.slice/crio-5d6e88047a69ac274c0d0dca01c89c9262f9729b1637e7fd35832b473852484d WatchSource:0}: Error finding container 5d6e88047a69ac274c0d0dca01c89c9262f9729b1637e7fd35832b473852484d: Status 404 returned error can't find the container with id 5d6e88047a69ac274c0d0dca01c89c9262f9729b1637e7fd35832b473852484d
Oct 30 12:45:28.537898 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.537861    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dlv29" event={"ID":"b0eab2da-9812-424c-9be0-af423308d43c","Type":"ContainerStarted","Data":"5d6e88047a69ac274c0d0dca01c89c9262f9729b1637e7fd35832b473852484d"}
Oct 30 12:45:28.539563 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.539536    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k47jk" event={"ID":"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265","Type":"ContainerStarted","Data":"e40e6e313fb96b15b80462b99acbb31fae950e317e6f8ab1c791a83c13eba500"}
Oct 30 12:45:28.539685 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.539586    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k47jk" event={"ID":"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265","Type":"ContainerStarted","Data":"4eab01a9fa445d90b530a1373c3ffa1610fa98e01cba407c987b00827a6edc17"}
Oct 30 12:45:28.540563 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:28.540538    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rm7hq" event={"ID":"6c8c3435-b6de-4aea-b596-970c9e17985c","Type":"ContainerStarted","Data":"b3c29c3db2ac1fc0c5bda0cdd5d028040453bfd2da80822c25c6946f7cf4f3bd"}
Oct 30 12:45:29.190682 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.190641    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"]
Oct 30 12:45:29.210848 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.210815    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:29.216922 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.216894    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"]
Oct 30 12:45:29.225833 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.225809    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2xmlk"]
Oct 30 12:45:29.227091 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.227064    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 12:45:29.227228 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.227102    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 12:45:29.227228 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.227118    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 12:45:29.227228 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.227148    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 12:45:29.227408 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.227320    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dqrs9\""
Oct 30 12:45:29.227408 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.227363    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 12:45:29.237176 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.237154    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"]
Oct 30 12:45:29.237410 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.237318    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.257770 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.257733    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.262969 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.262946    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"]
Oct 30 12:45:29.310428 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.310401    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 12:45:29.310595 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.310580    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 12:45:29.310683 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.310660    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5ldkz\""
Oct 30 12:45:29.310801 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.310701    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 12:45:29.310857 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.310806    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 12:45:29.310903 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.310859    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-njv68\""
Oct 30 12:45:29.326046 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.326020    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 12:45:29.326181 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.326086    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 12:45:29.347147 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347120    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-tls\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.347326 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347160    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afb494b7-75a0-400e-be16-74d8d6882460-metrics-client-ca\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.347326 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347186    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89288ada-24c5-46af-ad86-7d21935ca939-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:29.347326 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347212    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.347326 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347242    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x2pz\" (UniqueName: \"kubernetes.io/projected/afb494b7-75a0-400e-be16-74d8d6882460-kube-api-access-9x2pz\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.347326 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpcj9\" (UniqueName: \"kubernetes.io/projected/89288ada-24c5-46af-ad86-7d21935ca939-kube-api-access-mpcj9\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:29.347326 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347298    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9b17184b-71db-4557-ae32-062ae53a9d53-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.347326 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4zvz\" (UniqueName: \"kubernetes.io/projected/9b17184b-71db-4557-ae32-062ae53a9d53-kube-api-access-b4zvz\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.347665 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347348    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b17184b-71db-4557-ae32-062ae53a9d53-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.347665 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347376    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.347665 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347399    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-accelerators-collector-config\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.347665 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347424    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-wtmp\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.347665 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347441    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-textfile\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.347665 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-sys\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.347665 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.347665 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89288ada-24c5-46af-ad86-7d21935ca939-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:29.347665 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347530    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-root\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.347665 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347547    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.347665 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.347568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/89288ada-24c5-46af-ad86-7d21935ca939-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:29.448172 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.448172 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448135    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89288ada-24c5-46af-ad86-7d21935ca939-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:29.448172 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-root\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.448468 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448191    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.448468 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448226    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/89288ada-24c5-46af-ad86-7d21935ca939-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:29.448468 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-tls\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.448468 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afb494b7-75a0-400e-be16-74d8d6882460-metrics-client-ca\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.448468 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89288ada-24c5-46af-ad86-7d21935ca939-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:29.448468 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.448468 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448379    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x2pz\" (UniqueName: \"kubernetes.io/projected/afb494b7-75a0-400e-be16-74d8d6882460-kube-api-access-9x2pz\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.448468 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448405    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpcj9\" (UniqueName: \"kubernetes.io/projected/89288ada-24c5-46af-ad86-7d21935ca939-kube-api-access-mpcj9\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:29.448468 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9b17184b-71db-4557-ae32-062ae53a9d53-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.448468 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4zvz\" (UniqueName: \"kubernetes.io/projected/9b17184b-71db-4557-ae32-062ae53a9d53-kube-api-access-b4zvz\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.448931 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b17184b-71db-4557-ae32-062ae53a9d53-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.448931 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.448931 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-accelerators-collector-config\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.448931 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-wtmp\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.448931 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448589    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-textfile\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.448931 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-sys\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.448931 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448722    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-sys\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.449202 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.448932    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-root\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.449497 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.449462    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-wtmp\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.449621 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.449506    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-accelerators-collector-config\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.449671 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.449625    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b17184b-71db-4557-ae32-062ae53a9d53-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.449711 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.449679    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9b17184b-71db-4557-ae32-062ae53a9d53-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.450112 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.450077    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.450445 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.450422    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89288ada-24c5-46af-ad86-7d21935ca939-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:29.450525 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.450511    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-textfile\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.450689 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.450667    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afb494b7-75a0-400e-be16-74d8d6882460-metrics-client-ca\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.452826 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.452798    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.453237 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.453192    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-tls\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.453756 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.453736    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/89288ada-24c5-46af-ad86-7d21935ca939-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:29.454768 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.454710    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.454768 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.454721    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.455855 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.455830    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89288ada-24c5-46af-ad86-7d21935ca939-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:29.476082 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.476059    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x2pz\" (UniqueName: \"kubernetes.io/projected/afb494b7-75a0-400e-be16-74d8d6882460-kube-api-access-9x2pz\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.476465 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.476442    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpcj9\" (UniqueName: \"kubernetes.io/projected/89288ada-24c5-46af-ad86-7d21935ca939-kube-api-access-mpcj9\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:29.477004 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.476982    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4zvz\" (UniqueName: \"kubernetes.io/projected/9b17184b-71db-4557-ae32-062ae53a9d53-kube-api-access-b4zvz\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.558333 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.558107    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:29.614594 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.614365    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:29.665038 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.665007    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:29.878969 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.878937    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"]
Oct 30 12:45:29.885027 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:45:29.884997    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89288ada_24c5_46af_ad86_7d21935ca939.slice/crio-9a5a551bf44c18777ce165235fae8247aea1e9c746b181bc88d8b0b43821c2d4 WatchSource:0}: Error finding container 9a5a551bf44c18777ce165235fae8247aea1e9c746b181bc88d8b0b43821c2d4: Status 404 returned error can't find the container with id 9a5a551bf44c18777ce165235fae8247aea1e9c746b181bc88d8b0b43821c2d4
Oct 30 12:45:29.975561 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:29.975310    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"]
Oct 30 12:45:29.982724 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:45:29.982697    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b17184b_71db_4557_ae32_062ae53a9d53.slice/crio-2688554654a541a3f1221350f62a06f8fdd4a03976f114be264eee10b407512f WatchSource:0}: Error finding container 2688554654a541a3f1221350f62a06f8fdd4a03976f114be264eee10b407512f: Status 404 returned error can't find the container with id 2688554654a541a3f1221350f62a06f8fdd4a03976f114be264eee10b407512f
Oct 30 12:45:30.551445 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.551345    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k47jk" event={"ID":"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265","Type":"ContainerStarted","Data":"601daa38701820b32d34defd8ce0331b9d856845120611db62d9ab14f60da093"}
Oct 30 12:45:30.553977 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.553884    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t" event={"ID":"9b17184b-71db-4557-ae32-062ae53a9d53","Type":"ContainerStarted","Data":"2688554654a541a3f1221350f62a06f8fdd4a03976f114be264eee10b407512f"}
Oct 30 12:45:30.556956 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.556920    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2xmlk" event={"ID":"afb494b7-75a0-400e-be16-74d8d6882460","Type":"ContainerStarted","Data":"eebaba938b60e759e94218534a845423109b6af2546f7607e8f86f205420d311"}
Oct 30 12:45:30.560326 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.560249    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw" event={"ID":"89288ada-24c5-46af-ad86-7d21935ca939","Type":"ContainerStarted","Data":"c0f4f3cd81bec2867c16cee94f37bbb256b2ab686a585f92e29af19e4b33393d"}
Oct 30 12:45:30.560598 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.560512    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw" event={"ID":"89288ada-24c5-46af-ad86-7d21935ca939","Type":"ContainerStarted","Data":"3964066ece597feb5c719be1b61629c119025f35d6b902438901e5af132adb75"}
Oct 30 12:45:30.560598 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.560561    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw" event={"ID":"89288ada-24c5-46af-ad86-7d21935ca939","Type":"ContainerStarted","Data":"9a5a551bf44c18777ce165235fae8247aea1e9c746b181bc88d8b0b43821c2d4"}
Oct 30 12:45:30.633995 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.632714    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d587d759d-h9f5b"]
Oct 30 12:45:30.636414 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.635118    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.649425 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.649401    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 12:45:30.649584 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.649539    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jpdrb\""
Oct 30 12:45:30.650122 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.650089    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 12:45:30.651155 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.650600    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 12:45:30.651155 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.650737    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 12:45:30.651155 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.650828    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 12:45:30.651155 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.650938    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 12:45:30.651155 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.651029    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 12:45:30.672941 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.672908    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 12:45:30.673747 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.673719    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d587d759d-h9f5b"]
Oct 30 12:45:30.693012 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.692983    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:45:30.695940 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.695918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.731512 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.731424    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:45:30.749989 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.749591    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 12:45:30.749989 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.749691    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 12:45:30.749989 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.749710    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5q4kn\""
Oct 30 12:45:30.749989 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.749755    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 12:45:30.749989 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.749793    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 12:45:30.749989 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.749854    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 12:45:30.749989 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.749862    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 12:45:30.749989 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.749892    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 12:45:30.749989 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.749937    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 12:45:30.752354 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.752313    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 12:45:30.764631 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.764306    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.764631 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.764341    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79d36201-724a-4cea-be2b-2e23dbf45682-tmp\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.764631 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.764372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.764631 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.764394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.764631 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.764420    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6rml\" (UniqueName: \"kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.764631 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.764437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.764631 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.764471    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.764631 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.764501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.865321 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865216    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.865321 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865255    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cljmm\" (UniqueName: \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-kube-api-access-cljmm\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.865321 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865308    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.865502 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865337    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.865502 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865380    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.865502 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865407    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79d36201-724a-4cea-be2b-2e23dbf45682-tmp\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.865502 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.865502 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865468    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.865502 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865498    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.865744 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.865744 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rml\" (UniqueName: \"kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.865744 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-config-volume\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.865744 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.865744 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.865744 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.865744 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865696    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-config-out\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.865744 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-web-config\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.866105 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.866105 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865779    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.866105 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.866105 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.865837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.867100 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.867074    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.867880 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.867860    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.868736 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.868672    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79d36201-724a-4cea-be2b-2e23dbf45682-tmp\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.868887 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.868860    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.869317 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.869261    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.872784 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.872736    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.872784 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.872759    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.890856 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.890826    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6rml\" (UniqueName: \"kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.949705 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.949674    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:30.966210 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.966166    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cljmm\" (UniqueName: \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-kube-api-access-cljmm\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.966210 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.966202    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.966376 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.966220    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.966376 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.966267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.966376 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.966311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.966376 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.966351    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-config-volume\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.966571 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.966381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.966571 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.966437    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.966571 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.966470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-config-out\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.966571 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.966498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-web-config\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.966571 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.966533    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.966571 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.966564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.966851 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.966595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.968203 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.967873    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.973224 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.972228    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.973224 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.973169    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.973397 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.973307    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.975448 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.975393    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.982101 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.981693    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-config-out\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.982101 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.982059    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.982899 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.982879    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.982997 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.982902    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.983101 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.983076    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-config-volume\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.988156 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.988111    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-web-config\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.988741 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.988706    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:30.995087 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:30.995066    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cljmm\" (UniqueName: \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-kube-api-access-cljmm\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:31.021406 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.021383    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:31.278888 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.278647    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d587d759d-h9f5b"]
Oct 30 12:45:31.284774 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:45:31.284742    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d36201_724a_4cea_be2b_2e23dbf45682.slice/crio-416452e21dafcd020d6cce7283ba268ad5811bcfa51e211b076baafe482a1e6e WatchSource:0}: Error finding container 416452e21dafcd020d6cce7283ba268ad5811bcfa51e211b076baafe482a1e6e: Status 404 returned error can't find the container with id 416452e21dafcd020d6cce7283ba268ad5811bcfa51e211b076baafe482a1e6e
Oct 30 12:45:31.378641 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.378596    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:45:31.385088 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:45:31.385027    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c94b3e_9026_4fba_9788_03f2d981b53b.slice/crio-93bd4adc1974ea9c1a5ddfb97845c697f9be6f6b2f38715a630afc2f75214529 WatchSource:0}: Error finding container 93bd4adc1974ea9c1a5ddfb97845c697f9be6f6b2f38715a630afc2f75214529: Status 404 returned error can't find the container with id 93bd4adc1974ea9c1a5ddfb97845c697f9be6f6b2f38715a630afc2f75214529
Oct 30 12:45:31.514560 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.513056    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"]
Oct 30 12:45:31.516883 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.516247    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.536133 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.535025    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 12:45:31.536133 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.535073    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 12:45:31.536133 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.535110    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 12:45:31.536133 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.535136    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-366snq849hmfp\""
Oct 30 12:45:31.536133 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.535218    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fs22w\""
Oct 30 12:45:31.536133 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.535760    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 12:45:31.536133 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.535977    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 12:45:31.543931 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.543835    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"]
Oct 30 12:45:31.568701 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.568665    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d587d759d-h9f5b" event={"ID":"79d36201-724a-4cea-be2b-2e23dbf45682","Type":"ContainerStarted","Data":"416452e21dafcd020d6cce7283ba268ad5811bcfa51e211b076baafe482a1e6e"}
Oct 30 12:45:31.571029 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.570997    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rm7hq" event={"ID":"6c8c3435-b6de-4aea-b596-970c9e17985c","Type":"ContainerStarted","Data":"f53f8a84004719d7c1f26b7406f9587bfdb13f90184117e6c909bd6d968143f0"}
Oct 30 12:45:31.571151 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.571050    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rm7hq" event={"ID":"6c8c3435-b6de-4aea-b596-970c9e17985c","Type":"ContainerStarted","Data":"a10d56bd1c99b6883847a1be24bf99cfc477bf9ecf4cefa8a1c0ef09d18b1677"}
Oct 30 12:45:31.571320 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.571296    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:31.572963 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.572939    2576 generic.go:356] "Generic (PLEG): container finished" podID="afb494b7-75a0-400e-be16-74d8d6882460" containerID="a03252c3e03967e9d31311987ac868d680e7f56ff8bacaba7d67166ca53ff5eb" exitCode=0
Oct 30 12:45:31.573061 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.573043    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2xmlk" event={"ID":"afb494b7-75a0-400e-be16-74d8d6882460","Type":"ContainerDied","Data":"a03252c3e03967e9d31311987ac868d680e7f56ff8bacaba7d67166ca53ff5eb"}
Oct 30 12:45:31.577733 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.577632    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerStarted","Data":"93bd4adc1974ea9c1a5ddfb97845c697f9be6f6b2f38715a630afc2f75214529"}
Oct 30 12:45:31.609986 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.609937    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-rm7hq" podStartSLOduration=2.103872995 podStartE2EDuration="4.609917601s" totalImagesPullingTime="2.506044606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:31.60926266 +0000 UTC m=+44.821557191" watchObservedRunningTime="2026-10-30 12:45:31.609917601 +0000 UTC m=+44.822212134"
Oct 30 12:45:31.673713 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.673337    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.676525 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.674861    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfvck\" (UniqueName: \"kubernetes.io/projected/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-kube-api-access-sfvck\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.676525 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.674939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.676525 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.675025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.676525 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.675102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-tls\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.676525 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.675128    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-tmp\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.676525 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.675143    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-grpc-tls\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.676525 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.675166    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.676525 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.675193    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-metrics-client-ca\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.777153 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.775968    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-metrics-client-ca\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.777153 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.776006    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.777153 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.776049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfvck\" (UniqueName: \"kubernetes.io/projected/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-kube-api-access-sfvck\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.777153 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.776070    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.777153 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.776102    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.777153 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.776135    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-tls\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.777153 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.776156    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-tmp\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.777153 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.776171    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-grpc-tls\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.777153 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.776187    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.779669 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.778113    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-metrics-client-ca\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.779669 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.778452    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-tmp\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.781056 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.781004    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.781405 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.781366    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.782445 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.782428    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-grpc-tls\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.786624 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.786237    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.786624 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.786592    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.787726 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.787685    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-tls\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.804693 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.804664    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfvck\" (UniqueName: \"kubernetes.io/projected/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-kube-api-access-sfvck\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:31.852043 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:31.852015    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:32.227847 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.227666    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"]
Oct 30 12:45:32.233920 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:45:32.233888    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d4edeee_f5a5_497d_a53c_2a6c82a3fe85.slice/crio-18099047513f5ac201e212b67ae436d92c39eda836ffe1c9bebab71544c5cd31 WatchSource:0}: Error finding container 18099047513f5ac201e212b67ae436d92c39eda836ffe1c9bebab71544c5cd31: Status 404 returned error can't find the container with id 18099047513f5ac201e212b67ae436d92c39eda836ffe1c9bebab71544c5cd31
Oct 30 12:45:32.585426 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.585384    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k47jk" event={"ID":"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265","Type":"ContainerStarted","Data":"9104dede9c93fa4d316f30529c9366b22dd93fb226fa88019cd2a1398ba3266a"}
Oct 30 12:45:32.588835 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.588809    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t" event={"ID":"9b17184b-71db-4557-ae32-062ae53a9d53","Type":"ContainerStarted","Data":"6535cb296f8dc17280e1ea88170d9e823a7a3bf0e66fc877572ca676de162eef"}
Oct 30 12:45:32.588950 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.588859    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t" event={"ID":"9b17184b-71db-4557-ae32-062ae53a9d53","Type":"ContainerStarted","Data":"e3ffcf3942955275d6fe1a4eb80f55699443c47d91512080d992266f57684c68"}
Oct 30 12:45:32.588950 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.588878    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t" event={"ID":"9b17184b-71db-4557-ae32-062ae53a9d53","Type":"ContainerStarted","Data":"85e011384feba5ef878dc97674b69a4baf0c933175490c22af59f26e34abead2"}
Oct 30 12:45:32.593433 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.591319    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2xmlk" event={"ID":"afb494b7-75a0-400e-be16-74d8d6882460","Type":"ContainerStarted","Data":"7cebcccc9679b33e0af102b895415877df186b80e08cfae85243174f01507764"}
Oct 30 12:45:32.593433 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.591370    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2xmlk" event={"ID":"afb494b7-75a0-400e-be16-74d8d6882460","Type":"ContainerStarted","Data":"aaa8983000522297bd02e03b203d9a5fc3c0cec9206ea5fc1a5fa832eddb59e2"}
Oct 30 12:45:32.594243 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.593753    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw" event={"ID":"89288ada-24c5-46af-ad86-7d21935ca939","Type":"ContainerStarted","Data":"68e4b9124f338cede724aa60e6b98e55c6a7363448f4337e76082943a7b1977d"}
Oct 30 12:45:32.595904 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.595879    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-dlv29" event={"ID":"b0eab2da-9812-424c-9be0-af423308d43c","Type":"ContainerStarted","Data":"1f921084a7f9dcb010f1bbb8bcc505144d810c070070d101b521ed0af409d851"}
Oct 30 12:45:32.600121 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.600084    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8" event={"ID":"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85","Type":"ContainerStarted","Data":"18099047513f5ac201e212b67ae436d92c39eda836ffe1c9bebab71544c5cd31"}
Oct 30 12:45:32.667782 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.667722    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw" podStartSLOduration=2.104004554 podStartE2EDuration="3.667702032s" totalImagesPullingTime="1.563697478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:32.666859524 +0000 UTC m=+45.879154058" watchObservedRunningTime="2026-10-30 12:45:32.667702032 +0000 UTC m=+45.879996563"
Oct 30 12:45:32.668140 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.668102    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k47jk" podStartSLOduration=2.407598416 podStartE2EDuration="5.668094754s" totalImagesPullingTime="3.260496338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 12:45:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:32.626826292 +0000 UTC m=+45.839120841" watchObservedRunningTime="2026-10-30 12:45:32.668094754 +0000 UTC m=+45.880389283"
Oct 30 12:45:32.751007 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.750933    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2xmlk" podStartSLOduration=2.189208984 podStartE2EDuration="3.750912398s" totalImagesPullingTime="1.454162232s" totalInitContainerRuntime="107.541182ms" isStatefulPod=true podCreationTimestamp="2026-10-30 12:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:32.708845754 +0000 UTC m=+45.921140283" watchObservedRunningTime="2026-10-30 12:45:32.750912398 +0000 UTC m=+45.963206923"
Oct 30 12:45:32.792903 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.792308    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-dlv29" podStartSLOduration=1.726581599 podStartE2EDuration="5.792292023s" totalImagesPullingTime="4.065710424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:32.75062536 +0000 UTC m=+45.962919891" watchObservedRunningTime="2026-10-30 12:45:32.792292023 +0000 UTC m=+46.004586549"
Oct 30 12:45:32.792903 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:32.792689    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t" podStartSLOduration=1.792282506 podStartE2EDuration="3.792684405s" totalImagesPullingTime="2.000401899s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:32.791288852 +0000 UTC m=+46.003583378" watchObservedRunningTime="2026-10-30 12:45:32.792684405 +0000 UTC m=+46.004978935"
Oct 30 12:45:33.607933 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.607892    2576 generic.go:356] "Generic (PLEG): container finished" podID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerID="09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6" exitCode=0
Oct 30 12:45:33.609293 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.609249    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerDied","Data":"09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6"}
Oct 30 12:45:33.721641 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.721591    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-b6fc5f989-jm9qh"]
Oct 30 12:45:33.724479 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.724453    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.739549 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.739522    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 12:45:33.739699 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.739650    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 12:45:33.739943 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.739915    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 12:45:33.739943 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.739935    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 12:45:33.740108 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.739941    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xx2bf\""
Oct 30 12:45:33.740108 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.739975    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dtr69qhp6drph\""
Oct 30 12:45:33.745631 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.745611    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b6fc5f989-jm9qh"]
Oct 30 12:45:33.811082 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.811044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-secret-metrics-server-tls\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.811254 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.811093    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-client-ca-bundle\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.811254 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.811130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/963f75f5-2539-456a-ae78-267d5b002bab-audit-log\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.811254 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.811158    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w7lh\" (UniqueName: \"kubernetes.io/projected/963f75f5-2539-456a-ae78-267d5b002bab-kube-api-access-8w7lh\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.811254 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.811205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/963f75f5-2539-456a-ae78-267d5b002bab-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.811254 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.811249    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/963f75f5-2539-456a-ae78-267d5b002bab-metrics-server-audit-profiles\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.811540 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.811312    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-secret-metrics-server-client-certs\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.912261 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.912179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-client-ca-bundle\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.912261 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.912238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/963f75f5-2539-456a-ae78-267d5b002bab-audit-log\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.912500 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.912288    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w7lh\" (UniqueName: \"kubernetes.io/projected/963f75f5-2539-456a-ae78-267d5b002bab-kube-api-access-8w7lh\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.912500 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.912346    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/963f75f5-2539-456a-ae78-267d5b002bab-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.912500 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.912390    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/963f75f5-2539-456a-ae78-267d5b002bab-metrics-server-audit-profiles\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.912500 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.912431    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-secret-metrics-server-client-certs\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.912685 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.912503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-secret-metrics-server-tls\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.912738 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.912699    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/963f75f5-2539-456a-ae78-267d5b002bab-audit-log\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.913101 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.913080    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/963f75f5-2539-456a-ae78-267d5b002bab-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.913654 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.913626    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/963f75f5-2539-456a-ae78-267d5b002bab-metrics-server-audit-profiles\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.915520 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.915496    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-client-ca-bundle\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.915662 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.915645    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-secret-metrics-server-tls\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.915834 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.915811    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-secret-metrics-server-client-certs\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:33.935659 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:33.935626    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w7lh\" (UniqueName: \"kubernetes.io/projected/963f75f5-2539-456a-ae78-267d5b002bab-kube-api-access-8w7lh\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:34.038653 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.038351    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:34.066754 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.066692    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"]
Oct 30 12:45:34.071660 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.071635    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:45:34.090624 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.090587    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-s2l2p\""
Oct 30 12:45:34.090783 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.090663    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 12:45:34.092169 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.092127    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"]
Oct 30 12:45:34.114859 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.114826    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35840c58-b7a6-438e-aed5-6c0d9764a793-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-krk2q\" (UID: \"35840c58-b7a6-438e-aed5-6c0d9764a793\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:45:34.114984 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.114874    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87bm\" (UniqueName: \"kubernetes.io/projected/35840c58-b7a6-438e-aed5-6c0d9764a793-kube-api-access-b87bm\") pod \"monitoring-plugin-77f4f76594-krk2q\" (UID: \"35840c58-b7a6-438e-aed5-6c0d9764a793\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:45:34.216132 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.216098    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35840c58-b7a6-438e-aed5-6c0d9764a793-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-krk2q\" (UID: \"35840c58-b7a6-438e-aed5-6c0d9764a793\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:45:34.216264 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.216139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87bm\" (UniqueName: \"kubernetes.io/projected/35840c58-b7a6-438e-aed5-6c0d9764a793-kube-api-access-b87bm\") pod \"monitoring-plugin-77f4f76594-krk2q\" (UID: \"35840c58-b7a6-438e-aed5-6c0d9764a793\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:45:34.219309 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.219257    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35840c58-b7a6-438e-aed5-6c0d9764a793-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-krk2q\" (UID: \"35840c58-b7a6-438e-aed5-6c0d9764a793\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:45:34.239155 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.239090    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87bm\" (UniqueName: \"kubernetes.io/projected/35840c58-b7a6-438e-aed5-6c0d9764a793-kube-api-access-b87bm\") pod \"monitoring-plugin-77f4f76594-krk2q\" (UID: \"35840c58-b7a6-438e-aed5-6c0d9764a793\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:45:34.371374 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.371009    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-b6fc5f989-jm9qh"]
Oct 30 12:45:34.375756 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:45:34.375705    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963f75f5_2539_456a_ae78_267d5b002bab.slice/crio-bef9665aa0a5c505102fa3df13ce1b0e6f5acff3fea328a836dc20a91e4f4525 WatchSource:0}: Error finding container bef9665aa0a5c505102fa3df13ce1b0e6f5acff3fea328a836dc20a91e4f4525: Status 404 returned error can't find the container with id bef9665aa0a5c505102fa3df13ce1b0e6f5acff3fea328a836dc20a91e4f4525
Oct 30 12:45:34.384978 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.384954    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:45:34.613663 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.613583    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh" event={"ID":"963f75f5-2539-456a-ae78-267d5b002bab","Type":"ContainerStarted","Data":"bef9665aa0a5c505102fa3df13ce1b0e6f5acff3fea328a836dc20a91e4f4525"}
Oct 30 12:45:34.703831 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:34.703754    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"]
Oct 30 12:45:34.707650 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:45:34.707619    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35840c58_b7a6_438e_aed5_6c0d9764a793.slice/crio-7824adac02dc9da6019a262bbfff2cae0b9176774c857f6edd152adbd477223b WatchSource:0}: Error finding container 7824adac02dc9da6019a262bbfff2cae0b9176774c857f6edd152adbd477223b: Status 404 returned error can't find the container with id 7824adac02dc9da6019a262bbfff2cae0b9176774c857f6edd152adbd477223b
Oct 30 12:45:35.619442 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:35.618877    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q" event={"ID":"35840c58-b7a6-438e-aed5-6c0d9764a793","Type":"ContainerStarted","Data":"7824adac02dc9da6019a262bbfff2cae0b9176774c857f6edd152adbd477223b"}
Oct 30 12:45:35.621965 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:35.621937    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d587d759d-h9f5b" event={"ID":"79d36201-724a-4cea-be2b-2e23dbf45682","Type":"ContainerStarted","Data":"69ce50314b5f38bc1c6bae178174c5cc37252fd6f32432c139505b2fd03b915c"}
Oct 30 12:45:35.632502 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:35.632340    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8" event={"ID":"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85","Type":"ContainerStarted","Data":"34e94d006dc5d2534f4e0f722daf3731c537d81d83eadf3598cb842c9a89c996"}
Oct 30 12:45:35.632502 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:35.632417    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8" event={"ID":"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85","Type":"ContainerStarted","Data":"f96ed76970f2e17b2903710a47e43f5d5117edb4b845c970457400c42ece665c"}
Oct 30 12:45:35.632502 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:35.632456    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8" event={"ID":"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85","Type":"ContainerStarted","Data":"6dbfff56272937e624d542491d8c3611ec15b6919118e0794de6b06eedb7d849"}
Oct 30 12:45:35.637546 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:35.637341    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerStarted","Data":"6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb"}
Oct 30 12:45:35.637546 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:35.637409    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerStarted","Data":"e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218"}
Oct 30 12:45:35.637546 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:35.637450    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerStarted","Data":"4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747"}
Oct 30 12:45:35.637546 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:35.637486    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerStarted","Data":"087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494"}
Oct 30 12:45:35.662979 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:35.662913    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7d587d759d-h9f5b" podStartSLOduration=2.110512847 podStartE2EDuration="5.662896308s" totalImagesPullingTime="3.552383461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:35.662649172 +0000 UTC m=+48.874943722" watchObservedRunningTime="2026-10-30 12:45:35.662896308 +0000 UTC m=+48.875190842"
Oct 30 12:45:36.226598 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.226566    2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:45:36.232821 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.232790    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.246564 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.246540    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 12:45:36.246564 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.246540    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 12:45:36.246740 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.246576    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dcr46kt5lt1rp\""
Oct 30 12:45:36.246740 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.246647    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 12:45:36.246740 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.246654    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 12:45:36.247779 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.247758    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 12:45:36.257044 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.257023    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 12:45:36.258080 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.257377    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 12:45:36.258080 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.257503    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 12:45:36.258080 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.257569    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 12:45:36.258080 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.257597    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 12:45:36.258080 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.257693    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gxtbq\""
Oct 30 12:45:36.258080 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.257837    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 12:45:36.258080 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.257889    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 12:45:36.269554 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.269533    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 12:45:36.282212 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.280375    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 12:45:36.282212 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.282137    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:45:36.341666 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.341638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-web-config\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.341800 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.341695    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-config\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.341800 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.341771    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.341870 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.341806    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.341906 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.341874    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.341944 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.341907    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.341944 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.341924    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.342024 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.341945    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.342052 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.342030    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.342081 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.342052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.342145 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.342127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.342178 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.342162    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.342210 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.342183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.342239 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.342210    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-789l8\" (UniqueName: \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-kube-api-access-789l8\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.342297 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.342244    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-config-out\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.342337 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.342306    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.342369 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.342338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.342369 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.342355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.342421 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.342374    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.342421 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.342406    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.442811 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.442789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.442907 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.442824    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-web-config\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.442907 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.442850    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-config\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443016 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.442944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443016 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.442978    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443304 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443363 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443418 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443418 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443523 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443448    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443523 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443458    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443523 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443471    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443662 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443662 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443662 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443592    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443826 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-789l8\" (UniqueName: \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-kube-api-access-789l8\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443826 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443613    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443826 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443727    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443826 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443729    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-config-out\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.443826 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.444057 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.444057 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443855    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.444057 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.443888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.444602 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.444578    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.445161 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.445137    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.445621 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.445596    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.447448 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.447320    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.447448 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.447384    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-config-out\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.447633 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.447514    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.447633 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.447526    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.449151 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.449128    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.450248 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.450224    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.450951 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.450789    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.450951 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.450905    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-config\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.450951 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.450918    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-web-config\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.450951 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.450935    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.450951 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.450946    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.451157 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.451141    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.452553 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.452535    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.465577 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.465554    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-789l8\" (UniqueName: \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-kube-api-access-789l8\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.563846 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.563819    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:36.644776 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.644677    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerStarted","Data":"4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb"}
Oct 30 12:45:36.644776 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.644770    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerStarted","Data":"7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378"}
Oct 30 12:45:36.646541 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.646515    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q" event={"ID":"35840c58-b7a6-438e-aed5-6c0d9764a793","Type":"ContainerStarted","Data":"6808867ae1282585b61ce734fdeb202a1c6a64dd939d9aa4a353edc3dd665ba6"}
Oct 30 12:45:36.646646 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.646617    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:45:36.649968 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.649942    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8" event={"ID":"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85","Type":"ContainerStarted","Data":"efca95a1e640a4d62a2134234c255d77e7b786bcd822aa95b09e514f1b8e666c"}
Oct 30 12:45:36.650064 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.649998    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8" event={"ID":"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85","Type":"ContainerStarted","Data":"2524dd77b47c1d418c5bb92b870f72d9fc70ebad6250232eb893936dde04ea5f"}
Oct 30 12:45:36.650064 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.650034    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8" event={"ID":"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85","Type":"ContainerStarted","Data":"e1c66ebb55c6cdca41833f986b4ad1fab41da64bb8f3dba43cde3d5df94be14d"}
Oct 30 12:45:36.650149 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.650088    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:36.651558 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.651537    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh" event={"ID":"963f75f5-2539-456a-ae78-267d5b002bab","Type":"ContainerStarted","Data":"e1ad30e5ee636e6e1fb9ae35a11b12f9dde6c94c1950ee721d455be0985b5941"}
Oct 30 12:45:36.652098 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.652081    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:45:36.700773 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.700719    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.132617012 podStartE2EDuration="6.700700262s" totalImagesPullingTime="3.555751084s" totalInitContainerRuntime="12.332166ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:36.699313353 +0000 UTC m=+49.911607878" watchObservedRunningTime="2026-10-30 12:45:36.700700262 +0000 UTC m=+49.912994795"
Oct 30 12:45:36.791622 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.791566    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8" podStartSLOduration=2.45389009 podStartE2EDuration="5.791551608s" totalImagesPullingTime="3.337661518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:36.750784482 +0000 UTC m=+49.963079013" watchObservedRunningTime="2026-10-30 12:45:36.791551608 +0000 UTC m=+50.003846147"
Oct 30 12:45:36.791825 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.791806    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh" podStartSLOduration=1.796564177 podStartE2EDuration="3.7918009s" totalImagesPullingTime="1.995236723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:36.790458248 +0000 UTC m=+50.002752779" watchObservedRunningTime="2026-10-30 12:45:36.7918009 +0000 UTC m=+50.004095429"
Oct 30 12:45:36.829582 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.829534    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q" podStartSLOduration=1.163181653 podStartE2EDuration="2.829518373s" totalImagesPullingTime="1.66633672s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:36.829167933 +0000 UTC m=+50.041462463" watchObservedRunningTime="2026-10-30 12:45:36.829518373 +0000 UTC m=+50.041812907"
Oct 30 12:45:36.887652 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:36.887623    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:45:36.892443 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:45:36.892415    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd457ae45_4f12_43bf_b6a5_634a29981a55.slice/crio-80a9b43884d5c96da6c6eafdf9963cd816b6a79de6a12ec8eb1b7cf364f5ab63 WatchSource:0}: Error finding container 80a9b43884d5c96da6c6eafdf9963cd816b6a79de6a12ec8eb1b7cf364f5ab63: Status 404 returned error can't find the container with id 80a9b43884d5c96da6c6eafdf9963cd816b6a79de6a12ec8eb1b7cf364f5ab63
Oct 30 12:45:37.656789 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:37.656755    2576 generic.go:356] "Generic (PLEG): container finished" podID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerID="af2b81871b51f98092e2ca59b7ad6cd7e58f4d8f62a86ba6b153408ca3a82817" exitCode=0
Oct 30 12:45:37.657169 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:37.656867    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerDied","Data":"af2b81871b51f98092e2ca59b7ad6cd7e58f4d8f62a86ba6b153408ca3a82817"}
Oct 30 12:45:37.657169 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:37.656943    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerStarted","Data":"80a9b43884d5c96da6c6eafdf9963cd816b6a79de6a12ec8eb1b7cf364f5ab63"}
Oct 30 12:45:39.861502 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:39.861468    2576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d587d759d-h9f5b"]
Oct 30 12:45:40.686349 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:40.686245    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerStarted","Data":"b080e98d467a6afe0e2652e1b33796dbf26dccf93a6f10ab7221379bf076475d"}
Oct 30 12:45:40.686487 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:40.686376    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerStarted","Data":"401974b924fee06c62b687e370fa2a7b431adc89ab56e7709aee9ad6f2cae5e6"}
Oct 30 12:45:40.686487 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:40.686424    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerStarted","Data":"c82e4f116e13689a12702b09eb5c5c5d6ec9cbe795bb1f73d34d78e494d39c19"}
Oct 30 12:45:40.950125 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:40.950048    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:41.512970 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:41.512941    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:41.602413 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:41.602386    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:41.693344 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:41.693302    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerStarted","Data":"b4d075a3d8c7215deb70dea8c7e479734eec1b0243e05b1eaf47517f6b4581d7"}
Oct 30 12:45:41.693513 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:41.693374    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerStarted","Data":"e2e695ac1c14fb46d732d615cf850054de0e29b4d4a276fc10ab09ace1a2ca19"}
Oct 30 12:45:41.693513 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:41.693399    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerStarted","Data":"8fa292dd75a6c79f7f212a0d09e2fc4ff77fc7369910fb5e9c63494a2db58959"}
Oct 30 12:45:41.748782 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:41.748728    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9475910709999997 podStartE2EDuration="5.748712536s" totalImagesPullingTime="2.786467264s" totalInitContainerRuntime="14.654201ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:45:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:41.747324696 +0000 UTC m=+54.959619227" watchObservedRunningTime="2026-10-30 12:45:41.748712536 +0000 UTC m=+54.961007068"
Oct 30 12:45:42.663041 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:42.663013    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:46.564871 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:46.564841    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:52.999126 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:52.999073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:52.999548 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:52.999181    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:53.011488 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.011465    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:45:53.021910 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.021877    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:53.022122 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.022108    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:45:53.032517 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.032490    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:53.100145 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.100096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4cv\" (UniqueName: \"kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv\") pod \"network-check-target-7qcht\" (UID: \"34fc63c2-b395-4702-a2e4-134161c0d50c\") " pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:53.112742 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.112718    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:45:53.132334 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.132311    2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:45:53.142829 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.142802    2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq4cv\" (UniqueName: \"kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv\") pod \"network-check-target-7qcht\" (UID: \"34fc63c2-b395-4702-a2e4-134161c0d50c\") " pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:53.191692 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.191666    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w75sr\""
Oct 30 12:45:53.200246 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.200223    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:53.218192 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.218171    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zngfb\""
Oct 30 12:45:53.226432 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.226405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:53.228878 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.228853    2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5fhhc\""
Oct 30 12:45:53.236740 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.236710    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:53.499662 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.499628    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-79f2f"]
Oct 30 12:45:53.503360 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:45:53.503333    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408c5c75_c039_4dc1_89c7_6b0f4412aa59.slice/crio-49cf0d1aa23904c95d375634466822ecf2976144027b07593e3f1b1cefd43448 WatchSource:0}: Error finding container 49cf0d1aa23904c95d375634466822ecf2976144027b07593e3f1b1cefd43448: Status 404 returned error can't find the container with id 49cf0d1aa23904c95d375634466822ecf2976144027b07593e3f1b1cefd43448
Oct 30 12:45:53.540123 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.540095    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zfp4t"]
Oct 30 12:45:53.543871 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:45:53.543846    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825a2efb_6a8c_4bdd_bf89_cca7ee710f8e.slice/crio-81359d9d6eec84462ae650df8ff4d12d53d7e7ab668d7e0d50edd4b850960e4b WatchSource:0}: Error finding container 81359d9d6eec84462ae650df8ff4d12d53d7e7ab668d7e0d50edd4b850960e4b: Status 404 returned error can't find the container with id 81359d9d6eec84462ae650df8ff4d12d53d7e7ab668d7e0d50edd4b850960e4b
Oct 30 12:45:53.550212 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.550187    2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7qcht"]
Oct 30 12:45:53.554488 ip-10-0-141-232 kubenswrapper[2576]: W1030 12:45:53.554463    2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34fc63c2_b395_4702_a2e4_134161c0d50c.slice/crio-5868626f089311f71683d77ca857047ffca6c6148d29170c11553c58803f23b1 WatchSource:0}: Error finding container 5868626f089311f71683d77ca857047ffca6c6148d29170c11553c58803f23b1: Status 404 returned error can't find the container with id 5868626f089311f71683d77ca857047ffca6c6148d29170c11553c58803f23b1
Oct 30 12:45:53.737012 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.736977    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7qcht" event={"ID":"34fc63c2-b395-4702-a2e4-134161c0d50c","Type":"ContainerStarted","Data":"5868626f089311f71683d77ca857047ffca6c6148d29170c11553c58803f23b1"}
Oct 30 12:45:53.738152 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.738127    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfp4t" event={"ID":"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e","Type":"ContainerStarted","Data":"81359d9d6eec84462ae650df8ff4d12d53d7e7ab668d7e0d50edd4b850960e4b"}
Oct 30 12:45:53.739145 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:53.739126    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-79f2f" event={"ID":"408c5c75-c039-4dc1-89c7-6b0f4412aa59","Type":"ContainerStarted","Data":"49cf0d1aa23904c95d375634466822ecf2976144027b07593e3f1b1cefd43448"}
Oct 30 12:45:54.039035 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:54.039005    2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:54.039871 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:54.039846    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:55.751360 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:55.751323    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfp4t" event={"ID":"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e","Type":"ContainerStarted","Data":"4cbd162bbe052a7033d0cd1e8cedbd9a47cd2678416fc3d8a871f1e9dcfb2018"}
Oct 30 12:45:55.751821 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:55.751387    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zfp4t" event={"ID":"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e","Type":"ContainerStarted","Data":"2d2ef4a3e6173e7dd522ac6e046af4304d2af1d41bd1ea084b3c6be4fdc164cf"}
Oct 30 12:45:57.763263 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:57.763220    2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7qcht" event={"ID":"34fc63c2-b395-4702-a2e4-134161c0d50c","Type":"ContainerStarted","Data":"d71acc134472bc70117f9698131ac14a3b7cb2cbb5b1798207af0b4814276b3a"}
Oct 30 12:45:57.763741 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:57.763458    2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:57.808072 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:57.808007    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7qcht" podStartSLOduration=67.656624082 podStartE2EDuration="1m10.807989451s" totalImagesPullingTime="3.151365369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:57.806260128 +0000 UTC m=+71.018554672" watchObservedRunningTime="2026-10-30 12:45:57.807989451 +0000 UTC m=+71.020283980"
Oct 30 12:45:57.809063 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:57.809024    2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zfp4t" podStartSLOduration=69.496604975 podStartE2EDuration="1m10.809013751s" totalImagesPullingTime="1.312408776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:44:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:45:55.79099942 +0000 UTC m=+69.003293951" watchObservedRunningTime="2026-10-30 12:45:57.809013751 +0000 UTC m=+71.021308283"
Oct 30 12:45:58.054107 ip-10-0-141-232 kubenswrapper[2576]: E1030 12:45:58.054039    2576 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:45:58.361577 ip-10-0-141-232 kubenswrapper[2576]: I1030 12:45:58.361461    2576 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:45:58.361619 ip-10-0-141-232 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 12:45:58.371740 ip-10-0-141-232 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 12:45:58.371912 ip-10-0-141-232 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 12:45:58.372089 ip-10-0-141-232 systemd[1]: kubelet.service: Consumed 4.743s CPU time.
Oct 30 12:45:58.381772 ip-10-0-141-232 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 12:45:58.445920 ip-10-0-141-232 kubenswrapper[5893]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:45:58.445920 ip-10-0-141-232 kubenswrapper[5893]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 12:45:58.445920 ip-10-0-141-232 kubenswrapper[5893]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:45:58.445920 ip-10-0-141-232 kubenswrapper[5893]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 12:45:58.448043 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.447962    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:58.448043 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448043    5893 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:58.448043 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448047    5893 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448050    5893 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448053    5893 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448056    5893 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448059    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448061    5893 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448064    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448067    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448069    5893 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448074    5893 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448078    5893 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448081    5893 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448084    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448087    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448090    5893 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448093    5893 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448096    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448098    5893 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448101    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:58.448137 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448104    5893 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448107    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448109    5893 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448112    5893 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448115    5893 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448117    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448120    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448128    5893 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448131    5893 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448133    5893 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448136    5893 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448138    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448141    5893 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448144    5893 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448147    5893 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448150    5893 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448153    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448156    5893 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448158    5893 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448161    5893 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:58.448627 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448164    5893 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448166    5893 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448169    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448171    5893 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448174    5893 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448176    5893 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448179    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448181    5893 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448184    5893 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448186    5893 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448189    5893 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448192    5893 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448194    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448197    5893 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448200    5893 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448202    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448206    5893 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448209    5893 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448212    5893 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448215    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:58.449129 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448218    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448220    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448223    5893 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448225    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448228    5893 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448230    5893 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448233    5893 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448235    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448238    5893 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448240    5893 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448244    5893 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448246    5893 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448249    5893 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448252    5893 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448254    5893 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448256    5893 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448259    5893 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448284    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448288    5893 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448291    5893 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:58.449646 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448293    5893 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448296    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448299    5893 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448302    5893 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448305    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448307    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448311    5893 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448315    5893 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448318    5893 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448322    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448324    5893 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448327    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448329    5893 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448332    5893 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448334    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448336    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448339    5893 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448341    5893 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448344    5893 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:58.450132 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448346    5893 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448349    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448351    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448354    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448356    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.448359    5893 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449177    5893 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449185    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449189    5893 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449193    5893 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449198    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449205    5893 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449208    5893 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449211    5893 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449214    5893 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449217    5893 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449219    5893 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449222    5893 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449225    5893 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:58.450676 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449227    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449230    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449232    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449235    5893 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449237    5893 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449240    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449243    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449245    5893 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449248    5893 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449250    5893 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449253    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449255    5893 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449258    5893 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449260    5893 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449263    5893 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449265    5893 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449293    5893 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449297    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449299    5893 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449302    5893 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:58.451200 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449305    5893 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449307    5893 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449310    5893 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449313    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449315    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449318    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449321    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449324    5893 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449326    5893 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449329    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449331    5893 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449334    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449336    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449338    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449341    5893 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449343    5893 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449346    5893 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449348    5893 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449351    5893 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449353    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:58.451721 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449355    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449358    5893 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449360    5893 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449363    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449365    5893 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449368    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449370    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449373    5893 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449376    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449378    5893 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449381    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449383    5893 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449386    5893 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449389    5893 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449392    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449394    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449397    5893 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449399    5893 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449402    5893 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449405    5893 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:58.452225 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449407    5893 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449411    5893 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449413    5893 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449415    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449418    5893 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449420    5893 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449423    5893 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449426    5893 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449428    5893 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449431    5893 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449433    5893 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449436    5893 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449439    5893 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449441    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449443    5893 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449446    5893 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449449    5893 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449451    5893 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449454    5893 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449456    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:58.452751 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449459    5893 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449461    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449465    5893 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449469    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449472    5893 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449474    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449477    5893 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449480    5893 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449482    5893 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449486    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449489    5893 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449491    5893 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449494    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449975    5893 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449980    5893 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449984    5893 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449988    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449990    5893 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449993    5893 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:58.453240 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449996    5893 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.449999    5893 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450002    5893 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450004    5893 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450007    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450010    5893 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450012    5893 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450015    5893 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450017    5893 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450020    5893 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450022    5893 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450025    5893 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450028    5893 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450030    5893 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450032    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450035    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450037    5893 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450040    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450042    5893 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:58.453711 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450045    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450047    5893 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450050    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450052    5893 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450054    5893 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450057    5893 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450060    5893 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450062    5893 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450065    5893 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450067    5893 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450070    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450073    5893 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450075    5893 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450077    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450080    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450083    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450086    5893 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450088    5893 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450091    5893 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:58.454188 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450093    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450095    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450098    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450100    5893 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450103    5893 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450106    5893 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450108    5893 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450110    5893 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450113    5893 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450115    5893 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450118    5893 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450120    5893 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450127    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450129    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450132    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450135    5893 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450137    5893 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450140    5893 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450143    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450146    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:58.454672 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450149    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450151    5893 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450154    5893 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450158    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450160    5893 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450163    5893 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450166    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450169    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450172    5893 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450174    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450177    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450179    5893 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450182    5893 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450184    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450187    5893 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450191    5893 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450195    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450197    5893 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450200    5893 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:58.455244 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450203    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450206    5893 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450208    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450211    5893 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450213    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450217    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450220    5893 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450222    5893 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450225    5893 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450229    5893 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450232    5893 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450236    5893 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450239    5893 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450249    5893 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450252    5893 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450254    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450259    5893 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450262    5893 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450265    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450285    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:58.455726 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450288    5893 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450291    5893 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.450293    5893 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450382    5893 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450394    5893 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450406    5893 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450411    5893 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450415    5893 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450419    5893 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450423    5893 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450427    5893 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450430    5893 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450433    5893 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450436    5893 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450439    5893 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450442    5893 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450445    5893 flags.go:64] FLAG: --cgroup-root=""
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450450    5893 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450453    5893 flags.go:64] FLAG: --client-ca-file=""
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450456    5893 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450458    5893 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450465    5893 flags.go:64] FLAG: --cluster-domain=""
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450468    5893 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450471    5893 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 12:45:58.456234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450474    5893 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450478    5893 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450487    5893 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450491    5893 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450496    5893 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450500    5893 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450502    5893 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450505    5893 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450508    5893 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450511    5893 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450514    5893 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450518    5893 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450522    5893 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450525    5893 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450527    5893 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450530    5893 flags.go:64] FLAG: --enable-server="true"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450533    5893 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450540    5893 flags.go:64] FLAG: --event-burst="100"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450543    5893 flags.go:64] FLAG: --event-qps="50"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450546    5893 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450549    5893 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450552    5893 flags.go:64] FLAG: --eviction-hard=""
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450556    5893 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450559    5893 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450562    5893 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 12:45:58.456849 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450566    5893 flags.go:64] FLAG: --eviction-soft=""
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450569    5893 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450572    5893 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450575    5893 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450578    5893 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450581    5893 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450583    5893 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450586    5893 flags.go:64] FLAG: --feature-gates=""
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450590    5893 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450593    5893 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450597    5893 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450601    5893 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450604    5893 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450607    5893 flags.go:64] FLAG: --help="false"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450610    5893 flags.go:64] FLAG: --hostname-override="ip-10-0-141-232.ec2.internal"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450613    5893 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450616    5893 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450619    5893 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450622    5893 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450625    5893 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450628    5893 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450631    5893 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450634    5893 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 12:45:58.457483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450636    5893 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450639    5893 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450642    5893 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450645    5893 flags.go:64] FLAG: --kube-reserved=""
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450648    5893 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450651    5893 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450654    5893 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450656    5893 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450659    5893 flags.go:64] FLAG: --lock-file=""
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450663    5893 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450666    5893 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450669    5893 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450678    5893 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450681    5893 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450685    5893 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450688    5893 flags.go:64] FLAG: --logging-format="text"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450690    5893 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450694    5893 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450697    5893 flags.go:64] FLAG: --manifest-url=""
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450700    5893 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450708    5893 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450711    5893 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450715    5893 flags.go:64] FLAG: --max-pods="110"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450718    5893 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450721    5893 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 12:45:58.458044 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450724    5893 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450726    5893 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450729    5893 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450732    5893 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450735    5893 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450742    5893 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450745    5893 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450749    5893 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450752    5893 flags.go:64] FLAG: --pod-cidr=""
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450754    5893 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450757    5893 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450760    5893 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450763    5893 flags.go:64] FLAG: --port="10250"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450766    5893 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450769    5893 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-078b9bbb98442f0de"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450772    5893 flags.go:64] FLAG: --qos-reserved=""
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450776    5893 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450779    5893 flags.go:64] FLAG: --register-node="true"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450782    5893 flags.go:64] FLAG: --register-with-taints=""
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450786    5893 flags.go:64] FLAG: --registry-burst="10"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450788    5893 flags.go:64] FLAG: --registry-qps="5"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450791    5893 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450794    5893 flags.go:64] FLAG: --reserved-memory=""
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450798    5893 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450801    5893 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450804    5893 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 12:45:58.458678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450806    5893 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450811    5893 flags.go:64] FLAG: --runonce="false"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450814    5893 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450818    5893 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450821    5893 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450823    5893 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450826    5893 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450829    5893 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450832    5893 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450835    5893 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450838    5893 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450841    5893 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450843    5893 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450846    5893 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450849    5893 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450852    5893 flags.go:64] FLAG: --system-cgroups=""
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450855    5893 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450860    5893 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450862    5893 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450865    5893 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450871    5893 flags.go:64] FLAG: --tls-min-version=""
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450874    5893 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450878    5893 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450881    5893 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450884    5893 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 12:45:58.459804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450887    5893 flags.go:64] FLAG: --v="2"
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450891    5893 flags.go:64] FLAG: --version="false"
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450896    5893 flags.go:64] FLAG: --vmodule=""
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450899    5893 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.450903    5893 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451030    5893 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451034    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451036    5893 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451041    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451044    5893 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451047    5893 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451050    5893 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451052    5893 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451055    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451058    5893 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451060    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451063    5893 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451065    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451068    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451070    5893 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:58.460437 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451073    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451075    5893 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451078    5893 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451080    5893 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451083    5893 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451085    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451087    5893 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451090    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451093    5893 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451096    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451099    5893 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451101    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451104    5893 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451108    5893 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451111    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451114    5893 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451116    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451119    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451121    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:58.460929 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451124    5893 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451128    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451130    5893 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451132    5893 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451135    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451138    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451141    5893 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451143    5893 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451146    5893 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451148    5893 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451151    5893 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451153    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451156    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451158    5893 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451160    5893 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451163    5893 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451165    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451168    5893 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451170    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:58.461468 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451173    5893 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451175    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451177    5893 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451181    5893 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451184    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451186    5893 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451189    5893 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451191    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451194    5893 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451196    5893 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451199    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451201    5893 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451204    5893 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451206    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451210    5893 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451213    5893 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451216    5893 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451219    5893 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451224    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:58.462063 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451227    5893 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451230    5893 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451233    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451236    5893 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451239    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451241    5893 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451244    5893 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451246    5893 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451248    5893 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451251    5893 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451253    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451256    5893 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451258    5893 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451261    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451263    5893 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451266    5893 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451286    5893 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451290    5893 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451292    5893 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451295    5893 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:58.462659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451297    5893 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451300    5893 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451302    5893 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451305    5893 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451308    5893 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451310    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451312    5893 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451315    5893 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451318    5893 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451321    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451323    5893 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451326    5893 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451329    5893 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451332    5893 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.451357    5893 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451395    5893 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:58.463187 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451399    5893 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451402    5893 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451405    5893 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451408    5893 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451410    5893 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451413    5893 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451415    5893 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451418    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451420    5893 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451423    5893 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451425    5893 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451428    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451430    5893 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451434    5893 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451437    5893 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451439    5893 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451441    5893 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451444    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451446    5893 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451449    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:58.463659 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451451    5893 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451454    5893 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451457    5893 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451460    5893 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451462    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451465    5893 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451467    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451470    5893 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451473    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451476    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451478    5893 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451481    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451483    5893 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451486    5893 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451488    5893 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451491    5893 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451493    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451496    5893 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451498    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451501    5893 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:58.464179 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451503    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451505    5893 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451508    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451510    5893 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451514    5893 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451518    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451521    5893 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451524    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451526    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451529    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451531    5893 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451534    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451536    5893 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451538    5893 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451541    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451543    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451547    5893 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451550    5893 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451552    5893 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:58.464836 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451555    5893 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451557    5893 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451560    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451563    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451565    5893 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451568    5893 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451570    5893 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451573    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451575    5893 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451577    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451580    5893 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451584    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451586    5893 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451589    5893 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451591    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451594    5893 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451596    5893 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451599    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451602    5893 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:58.465383 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451605    5893 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451607    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451610    5893 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451612    5893 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451615    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451618    5893 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451620    5893 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451623    5893 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451626    5893 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451630    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451633    5893 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451636    5893 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451638    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451641    5893 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451644    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451646    5893 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451649    5893 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451651    5893 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451654    5893 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:58.465845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451656    5893 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451659    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451661    5893 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451664    5893 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451666    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451670    5893 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451673    5893 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.451675    5893 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.455501    5893 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.455514    5893 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.455531    5893 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.455535    5893 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455591    5893 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455596    5893 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455599    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455602    5893 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455605    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455607    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455610    5893 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:58.466361 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455612    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455615    5893 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455617    5893 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455620    5893 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455623    5893 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455625    5893 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455628    5893 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455630    5893 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455633    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455636    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455638    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455643    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455645    5893 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455648    5893 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455651    5893 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455655    5893 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455658    5893 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455661    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455664    5893 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:58.466834 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455667    5893 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455669    5893 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455672    5893 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455674    5893 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455676    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455679    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455681    5893 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455684    5893 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455686    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455689    5893 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455692    5893 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455694    5893 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455697    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455699    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455701    5893 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455704    5893 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455707    5893 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455709    5893 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455712    5893 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:58.467477 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455714    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455717    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455721    5893 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455723    5893 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455726    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455737    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455743    5893 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455746    5893 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455749    5893 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455751    5893 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455754    5893 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455756    5893 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455759    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455761    5893 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455764    5893 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455767    5893 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455769    5893 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455772    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455774    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:58.467974 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455776    5893 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455779    5893 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455782    5893 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455784    5893 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455787    5893 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455789    5893 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455792    5893 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455794    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455797    5893 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455799    5893 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455802    5893 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455804    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455807    5893 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455809    5893 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455811    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455814    5893 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455817    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455819    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455822    5893 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455824    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:58.468480 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455827    5893 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455830    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455832    5893 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455834    5893 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455837    5893 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455839    5893 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455842    5893 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455844    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455847    5893 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455849    5893 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455852    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455854    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455857    5893 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455859    5893 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455862    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455864    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455867    5893 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455869    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455872    5893 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:58.468948 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455874    5893 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455878    5893 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.455881    5893 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.455909    5893 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456047    5893 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456052    5893 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456054    5893 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456058    5893 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456061    5893 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456063    5893 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456066    5893 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456069    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456071    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456074    5893 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456076    5893 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456079    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:58.469415 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456082    5893 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456084    5893 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456087    5893 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456090    5893 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456094    5893 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456097    5893 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456100    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456102    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456105    5893 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456108    5893 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456110    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456112    5893 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456115    5893 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456117    5893 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456120    5893 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456122    5893 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456124    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456127    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456129    5893 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:58.469797 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456132    5893 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456134    5893 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456136    5893 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456138    5893 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456141    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456144    5893 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456148    5893 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456152    5893 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456155    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456158    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456161    5893 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456164    5893 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456167    5893 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456169    5893 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456172    5893 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456174    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456177    5893 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456179    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456182    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:58.470249 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456184    5893 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456186    5893 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456189    5893 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456191    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456194    5893 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456196    5893 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456198    5893 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456201    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456207    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456210    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456212    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456215    5893 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456217    5893 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456220    5893 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456222    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456225    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456227    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456230    5893 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456233    5893 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:58.470778 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456235    5893 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456238    5893 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456241    5893 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456243    5893 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456246    5893 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456249    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456252    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456254    5893 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456257    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456259    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456262    5893 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456264    5893 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456267    5893 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456287    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456291    5893 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456294    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456297    5893 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456299    5893 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456302    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456304    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:58.471254 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456307    5893 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456311    5893 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456314    5893 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456316    5893 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456319    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456322    5893 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456324    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456327    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456329    5893 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456332    5893 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456334    5893 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456337    5893 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456339    5893 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456341    5893 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456344    5893 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456346    5893 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456349    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:58.471782 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.456372    5893 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456404    5893 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456409    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456413    5893 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456417    5893 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456421    5893 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456424    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456427    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456429    5893 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456432    5893 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456434    5893 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456437    5893 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456439    5893 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456442    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456444    5893 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456447    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456449    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456453    5893 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456456    5893 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 12:45:58.472397 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456458    5893 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456461    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456463    5893 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456466    5893 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456468    5893 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456471    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456473    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456476    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456478    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456480    5893 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456483    5893 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456485    5893 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456488    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456490    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456492    5893 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456495    5893 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456497    5893 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456500    5893 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456503    5893 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 12:45:58.473114 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456505    5893 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456508    5893 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456510    5893 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456513    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456515    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456518    5893 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456520    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456522    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456525    5893 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456527    5893 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456531    5893 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456535    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456538    5893 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456540    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456543    5893 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456545    5893 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456547    5893 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456550    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456552    5893 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456555    5893 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 12:45:58.473715 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456557    5893 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456560    5893 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456562    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456566    5893 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456568    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456570    5893 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456573    5893 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456575    5893 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456578    5893 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456580    5893 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456583    5893 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456585    5893 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456588    5893 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456590    5893 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456593    5893 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456595    5893 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456597    5893 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456600    5893 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456603    5893 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456605    5893 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 12:45:58.474432 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456607    5893 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456610    5893 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456614    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456617    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456620    5893 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456622    5893 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456625    5893 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456627    5893 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456630    5893 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456632    5893 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456636    5893 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456639    5893 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456642    5893 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456644    5893 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456648    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456651    5893 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456653    5893 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456655    5893 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456658    5893 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456660    5893 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 12:45:58.475237 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456663    5893 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456665    5893 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456668    5893 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456670    5893 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456673    5893 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456675    5893 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456678    5893 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456680    5893 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:45:58.456682    5893 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.456729    5893 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.457680    5893 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.457734    5893 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.457902    5893 server.go:1014] "Starting client certificate rotation"
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.458046    5893 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.458095    5893 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:39:47 +0000 UTC" deadline="2028-03-30 14:49:19.721634453 +0000 UTC"
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.458124    5893 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12410h3m21.263512369s"
Oct 30 12:45:58.475804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.458336    5893 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:45:58.476204 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.460713    5893 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 12:45:58.476204 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.462137    5893 log.go:25] "Validated CRI v1 runtime API"
Oct 30 12:45:58.476204 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.464083    5893 log.go:25] "Validated CRI v1 image API"
Oct 30 12:45:58.476204 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.464787    5893 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 12:45:58.476204 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.467083    5893 fs.go:135] Filesystem UUIDs: map[76dcbf4e-7fa3-485c-b477-0cf87d7bd7cb:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 89e1d28a-48d5-44c6-ac23-7c268b390908:/dev/nvme0n1p4]
Oct 30 12:45:58.476441 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.467106    5893 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02210ac7103cb8fee0c9d6922b3f774b7409cf3c3e38509722954e36eb165f47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02210ac7103cb8fee0c9d6922b3f774b7409cf3c3e38509722954e36eb165f47/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02cd5d46eed573ed4f4904ba91544ea05df502d39040ed3ff2df9a21f6043c72/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02cd5d46eed573ed4f4904ba91544ea05df502d39040ed3ff2df9a21f6043c72/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/12dc6cc351023a89b3b2c74b20fb99e2cf2e49a084ed619bb6247ff9a2fdc8c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/12dc6cc351023a89b3b2c74b20fb99e2cf2e49a084ed619bb6247ff9a2fdc8c0/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/152eef17a3f22f68eb64380aa1414873f87aa636e72173280703463843e98214/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/152eef17a3f22f68eb64380aa1414873f87aa636e72173280703463843e98214/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18099047513f5ac201e212b67ae436d92c39eda836ffe1c9bebab71544c5cd31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18099047513f5ac201e212b67ae436d92c39eda836ffe1c9bebab71544c5cd31/userdata/shm major:0 minor:320 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2688554654a541a3f1221350f62a06f8fdd4a03976f114be264eee10b407512f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2688554654a541a3f1221350f62a06f8fdd4a03976f114be264eee10b407512f/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d3d2dc6f41867ef363f9445d7796544e423b55896729f29a7eb077f8bc2f3c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d3d2dc6f41867ef363f9445d7796544e423b55896729f29a7eb077f8bc2f3c7/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/416452e21dafcd020d6cce7283ba268ad5811bcfa51e211b076baafe482a1e6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/416452e21dafcd020d6cce7283ba268ad5811bcfa51e211b076baafe482a1e6e/userdata/shm major:0 minor:265 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49cf0d1aa23904c95d375634466822ecf2976144027b07593e3f1b1cefd43448/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49cf0d1aa23904c95d375634466822ecf2976144027b07593e3f1b1cefd43448/userdata/shm major:0 minor:489 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4eab01a9fa445d90b530a1373c3ffa1610fa98e01cba407c987b00827a6edc17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4eab01a9fa445d90b530a1373c3ffa1610fa98e01cba407c987b00827a6edc17/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5868626f089311f71683d77ca857047ffca6c6148d29170c11553c58803f23b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5868626f089311f71683d77ca857047ffca6c6148d29170c11553c58803f23b1/userdata/shm major:0 minor:493 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5d6e88047a69ac274c0d0dca01c89c9262f9729b1637e7fd35832b473852484d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5d6e88047a69ac274c0d0dca01c89c9262f9729b1637e7fd35832b473852484d/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6fdc177b4754e39d1a1dabb5ddb43e599c64550858fca406030492b3a2072321/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6fdc177b4754e39d1a1dabb5ddb43e599c64550858fca406030492b3a2072321/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7824adac02dc9da6019a262bbfff2cae0b9176774c857f6edd152adbd477223b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7824adac02dc9da6019a262bbfff2cae0b9176774c857f6edd152adbd477223b/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80a9b43884d5c96da6c6eafdf9963cd816b6a79de6a12ec8eb1b7cf364f5ab63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80a9b43884d5c96da6c6eafdf9963cd816b6a79de6a12ec8eb1b7cf364f5ab63/userdata/shm major:0 minor:451 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81359d9d6eec84462ae650df8ff4d12d53d7e7ab668d7e0d50edd4b850960e4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81359d9d6eec84462ae650df8ff4d12d53d7e7ab668d7e0d50edd4b850960e4b/userdata/shm major:0 minor:491 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86db01e788f2fc0e7e4e20864659d62fbe73c296d6217476e2cbb5065b738b3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86db01e788f2fc0e7e4e20864659d62fbe73c296d6217476e2cbb5065b738b3a/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8704082ecaafee1761693042fddc9c101ad08d9f08f9e9af632856b129dfcc58/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8704082ecaafee1761693042fddc9c101ad08d9f08f9e9af632856b129dfcc58/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93bd4adc1974ea9c1a5ddfb97845c697f9be6f6b2f38715a630afc2f75214529/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93bd4adc1974ea9c1a5ddfb97845c697f9be6f6b2f38715a630afc2f75214529/userdata/shm major:0 minor:267 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a5a551bf44c18777ce165235fae8247aea1e9c746b181bc88d8b0b43821c2d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a5a551bf44c18777ce165235fae8247aea1e9c746b181bc88d8b0b43821c2d4/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a1bdce8d58370fdadfaf068b0920e59bb20f3e3f7d66e53d3a6646e9a2d8753e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a1bdce8d58370fdadfaf068b0920e59bb20f3e3f7d66e53d3a6646e9a2d8753e/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3c29c3db2ac1fc0c5bda0cdd5d028040453bfd2da80822c25c6946f7cf4f3bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3c29c3db2ac1fc0c5bda0cdd5d028040453bfd2da80822c25c6946f7cf4f3bd/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bef9665aa0a5c505102fa3df13ce1b0e6f5acff3fea328a836dc20a91e4f4525/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bef9665aa0a5c505102fa3df13ce1b0e6f5acff3fea328a836dc20a91e4f4525/userdata/shm major:0 minor:349 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce9500fc0e9de19d8eab6cfa84c7dcdf6aa309e4b785f46c662933584e330b50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce9500fc0e9de19d8eab6cfa84c7dcdf6aa309e4b785f46c662933584e330b50/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eebaba938b60e759e94218534a845423109b6af2546f7607e8f86f205420d311/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eebaba938b60e759e94218534a845423109b6af2546f7607e8f86f205420d311/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/34fc63c2-b395-4702-a2e4-134161c0d50c/volumes/kubernetes.io~projected/kube-api-access-gq4cv:{mountpoint:/var/lib/kubelet/pods/34fc63c2-b395-4702-a2e4-134161c0d50c/volumes/kubernetes.io~projected/kube-api-access-gq4cv major:0 minor:488 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35840c58-b7a6-438e-aed5-6c0d9764a793/volumes/kubernetes.io~projected/kube-api-access-b87bm:{mountpoint:/var/lib/kubelet/pods/35840c58-b7a6-438e-aed5-6c0d9764a793/volumes/kubernetes.io~projected/kube-api-access-b87bm major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35840c58-b7a6-438e-aed5-6c0d9764a793/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/35840c58-b7a6-438e-aed5-6c0d9764a793/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36c3843d-81b7-4554-80ab-f6fc5901f488/volumes/kubernetes.io~projected/kube-api-access-9whn4:{mountpoint:/var/lib/kubelet/pods/36c3843d-81b7-4554-80ab-f6fc5901f488/volumes/kubernetes.io~projected/kube-api-access-9whn4 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~projected/kube-api-access-sfvck:{mountpoint:/var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~projected/kube-api-access-sfvck major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:308 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/408c5c75-c039-4dc1-89c7-6b0f4412aa59/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/408c5c75-c039-4dc1-89c7-6b0f4412aa59/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:487 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/429e3a90-1868-4632-8631-5c0207978063/volumes/kubernetes.io~projected/kube-api-access-wm2ml:{mountpoint:/var/lib/kubelet/pods/429e3a90-1868-4632-8631-5c0207978063/volumes/kubernetes.io~projected/kube-api-access-wm2ml major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/52f5c091-40a2-46e8-9214-ddf02da33d0b/volumes/kubernetes.io~projected/kube-api-access-86mph:{mountpoint:/var/lib/kubelet/pods/52f5c091-40a2-46e8-9214-ddf02da33d0b/volumes/kubernetes.io~projected/kube-api-access-86mph major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volume-subpaths/web-config/alertmanager/9 major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volume-subpaths/web-config/config-reloader/6 major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~empty-dir/config-out major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~projected/kube-api-access-cljmm:{mountpoint:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~projected/kube-api-access-cljmm major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~projected/tls-assets major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/config-volume major:0 minor:269 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:268 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:273 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/web-config major:0 minor:275 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c8c3435-b6de-4aea-b596-970c9e17985c/volumes/kubernetes.io~projected/kube-api-access-rkwhq:{mountpoint:/var/lib/kubelet/pods/6c8c3435-b6de-4aea-b596-970c9e17985c/volumes/kubernetes.io~projected/kube-api-access-rkwhq major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c8c3435-b6de-4aea-b596-970c9e17985c/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/6c8c3435-b6de-4aea-b596-970c9e17985c/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79d36201-724a-4cea-be2b-2e23dbf45682/volumes/kubernetes.io~projected/kube-api-access-c6rml:{mountpoint:/var/lib/kubelet/pods/79d36201-724a-4cea-be2b-2e23dbf45682/volumes/kubernetes.io~projected/kube-api-access-c6rml major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79d36201-724a-4cea-be2b-2e23dbf45682/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/79d36201-724a-4cea-be2b-2e23dbf45682/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79d36201-724a-4cea-be2b-2e23dbf45682/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/79d36201-724a-4cea-be2b-2e23dbf45682/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e/volumes/kubernetes.io~projected/kube-api-access-lcxs7:{mountpoint:/var/lib/kubelet/pods/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e/volumes/kubernetes.io~projected/kube-api-access-lcxs7 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e/volumes/kubernetes.io~secret/metrics-certs major:0 minor:483 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89288ada-24c5-46af-ad86-7d21935ca939/volumes/kubernetes.io~projected/kube-api-access-mpcj9:{mountpoint:/var/lib/kubelet/pods/89288ada-24c5-46af-ad86-7d21935ca939/volumes/kubernetes.io~projected/kube-api-access-mpcj9 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89288ada-24c5-46af-ad86-7d21935ca939/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/89288ada-24c5-46af-ad86-7d21935ca939/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89288ada-24c5-46af-ad86-7d21935ca939/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/89288ada-24c5-46af-ad86-7d21935ca939/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/963f75f5-2539-456a-ae78-267d5b002bab/volumes/kubernetes.io~projected/kube-api-access-8w7lh:{mountpoint:/var/lib/kubelet/pods/963f75f5-2539-456a-ae78-267d5b002bab/volumes/kubernetes.io~projected/kube-api-access-8w7lh major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/963f75f5-2539-456a-ae78-267d5b002bab/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/963f75f5-2539-456a-ae78-267d5b002bab/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/963f75f5-2539-456a-ae78-267d5b002bab/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/963f75f5-2539-456a-ae78-267d5b002bab/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/963f75f5-2539-456a-ae78-267d5b002bab/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/963f75f5-2539-456a-ae78-267d5b002bab/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b17184b-71db-4557-ae32-062ae53a9d53/volumes/kubernetes.io~projected/kube-api-access-b4zvz:{mountpoint:/var/lib/kubelet/pods/9b17184b-71db-4557-ae32-062ae53a9d53/volumes/kubernetes.io~projected/kube-api-access-b4zvz major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b17184b-71db-4557-ae32-062ae53a9d53/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9b17184b-71db-4557-ae32-062ae53a9d53/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b17184b-71db-4557-ae32-062ae53a9d53/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/9b17184b-71db-4557-ae32-062ae53a9d53/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bbdf052-fa04-4e1c-983a-04b313a0cdfb/volumes/kubernetes.io~projected/kube-api-access-x8s98:{mountpoint:/var/lib/kubelet/pods/9bbdf052-fa04-4e1c-983a-04b313a0cdfb/volumes/kubernetes.io~projected/kube-api-access-x8s98 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8053a33-e1bf-43f4-b23b-34694e7ce042/volumes/kubernetes.io~projected/kube-api-access-sm9qn:{mountpoint:/var/lib/kubelet/pods/a8053a33-e1bf-43f4-b23b-34694e7ce042/volumes/kubernetes.io~projected/kube-api-access-sm9qn major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afb494b7-75a0-400e-be16-74d8d6882460/volumes/kubernetes.io~projected/kube-api-access-9x2pz:{mountpoint:/var/lib/kubelet/pods/afb494b7-75a0-400e-be16-74d8d6882460/volumes/kubernetes.io~projected/kube-api-access-9x2pz major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afb494b7-75a0-400e-be16-74d8d6882460/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/afb494b7-75a0-400e-be16-74d8d6882460/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/afb494b7-75a0-400e-be16-74d8d6882460/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/afb494b7-75a0-400e-be16-74d8d6882460/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0eab2da-9812-424c-9be0-af423308d43c/volumes/kubernetes.io~projected/kube-api-access-sllfq:{mountpoint:/var/lib/kubelet/pods/b0eab2da-9812-424c-9be0-af423308d43c/volumes/kubernetes.io~projected/kube-api-access-sllfq major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0eab2da-9812-424c-9be0-af423308d43c/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b0eab2da-9812-424c-9be0-af423308d43c/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba6439a7-7042-4edf-aa9c-6d33b168873d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ba6439a7-7042-4edf-aa9c-6d33b168873d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba6439a7-7042-4edf-aa9c-6d33b168873d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ba6439a7-7042-4edf-aa9c-6d33b168873d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba6439a7-7042-4edf-aa9c-6d33b168873d/volumes/kubernetes.io~projected/kube-api-access-skwdd:{mountpoint:/var/lib/kubelet/pods/ba6439a7-7042-4edf-aa9c-6d33b168873d/volumes/kubernetes.io~projected/kube-api-access-skwdd major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volume-subpaths/web-config/config-reloader/2 major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volume-subpaths/web-config/prometheus/15 major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~empty-dir/config-out major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~projected/kube-api-access-789l8:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~projected/kube-api-access-789l8 major:0 minor:450 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~projected/tls-assets major:0 minor:444 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/config major:0 minor:442 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:445 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:448 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:446 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:447 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:431 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/web-config major:0 minor:443 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dccd197b-ec88-445b-8de9-ff83832112bd/volumes/kubernetes.io~projected/kube-api-access-rlm6t:{mountpoint:/var/lib/kubelet/pods/dccd197b-ec88-445b-8de9-ff83832112bd/volumes/kubernetes.io~projected/kube-api-access-rlm6t major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6897566-da42-4b8d-ba85-7ec304624f2b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f6897566-da42-4b8d-ba85-7ec304624f2b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6897566-da42-4b8d-ba85-7ec304624f2b/volumes/kubernetes.io~projected/kube-api-access-dh9mp:{mountpoint:/var/lib/kubelet/pods/f6897566-da42-4b8d-ba85-7ec304624f2b/volumes/kubernetes.io~projected/kube-api-access-dh9mp major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6897566-da42-4b8d-ba85-7ec304624f2b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f6897566-da42-4b8d-ba85-7ec304624f2b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265/volumes/kubernetes.io~projected/kube-api-access-6vkrz:{mountpoint:/var/lib/kubelet/pods/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265/volumes/kubernetes.io~projected/kube-api-access-6vkrz major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1d03b04a6bafeb21752e9eea5563790e87cfbe28bb2120f9e3d252ce4b8c7c42/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/5e907b8f9f34c6891b67fcf2dbbdf51115587214c3ea35f507dda2da0136e425/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/3c6e27f0a8071dce652eb0908a8f7684f0fc624bb2dddadda86a6dc87c0cc198/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ef1d6ae832065c92b5eb108c7dbc0f7a7a0e44eb634ce249a41ab6e4eeccec04/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a11982a661cb876ad4ed7ca58aae709d402bed40034e635e025ab73fca5a454b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/cafc7c8ba29d8c5852f264b6c8ecafabcd586b4e24fbf3ca8476c3a34e78dc22/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e2c86ddf0ec92dde7ceb17121a80eba32ec22d788d86df78245237323b200d94/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/e68d9b264da11f7655bcced9e2e803676c748fb02463b53be4bc11442b1d5cac/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/13d3b20cd9a26e330de0733482ad3274a5c4be6ab77095dfe04dd935e4ea8e16/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/41df57d686534f9c9e4f1af3c3f3f032a2c16dae64f39890ddc4ec4d11e5728e/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/3a1dde2fca0e0025e1e4e3e0c94ab261c0b6a727d1b62a541b730c5f94804f49/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/e45c893d752e249b089bba2fd6221461532d328061da2e6b9b07bb3648b12901/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/e4b3522693fecf2783d6e98972ecc241a7e46d4083ba8a7088b8b14cab56d857/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/bc3026ef94eab36c5f03c7b8296a116ffc27fd049b8b765235c7b740bdd9666e/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/d83a0333f270f451dddc06058a074b7553bd95aac393f578a450d81589d2e5d1/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9fc5778ca7b17557cef2247281c0628b5d9aa89a624f7120d4bd8b064412eecd/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/70dde25f716a5ff8eedd4a125cdd5230bb8f24914b8084b27c53d5c5d2291941/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c442bf0bec0e237133f6b9a70bbefa94cc19963c8e2a39c7cd49029dcafa649a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f5971c7a8fec7efd3de516dd7abdf1110e8b72269ad2883f11c17bc41dfd3e5e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f748657e36524a4b5e14c719fe2507e3cfd8d52318973689042d91d56008ae31/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/76604280e247f02a156f0b301e7c4fca14cc9d90d0ebd93d5c40a8b6de90b39f/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/c3e29ae073a5146f0b59c2272b104f4247f860bbb58a81446fe969f4cedc0ef1/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/20f279555dea72725482782fef1354090ecb0dbcf0a1caa6db8a4e9de32d8d62/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/f8a789d0e468d3d9e4f4692f3ff63f6083d69dcdf8f8b7d0daead1da995c33e9/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/6f41ae5add33fdd443512515fdd09ec687eb92f6fb77b27949528987fd094926/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/c1b2759103ca61af230c854a79b69e6783a1a08ed2a0cb32d9090696e42306c9/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/7af355f0bd08452a94d57d5ed82572bbed497dd6635d5ea35ce4756aa28446ff/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/8a2689f2f2d03960583972031ae5742ae063148db3257334fa16b3eb43284441/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/24d14cbc7d063689129799036093600f193539c9f3e2a7219bf88b21396dd55f/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/dc907a9bf031133ab8bf3900473e32ab441531b35048eff5eb98fd2968f5564a/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/e63be7878f3f4420cd4099ddfdb370660e7e23e72c86e6391411903988acb657/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/df01d3c23bbe8e5c8ebeb74125c25ce03e55c0ed25c1eece186f241638aa4049/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/03e4a8ac79678f244ae935140af59237b25d77be6ddca9da3ddf624f2b248ce0/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/1b5cf4cc93de13d0075f26166f35cbbc021eba98f80ca664625dcd3405c63dab/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/32108877cd58321b5eade7c939c6647a86437066441b75d72adbe298b33a0d3c/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/163b1eb678007bd9b14495128b4aaed69b4efa349df15d5007a1f15c20ba75ff/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/a88ecc70f9f8d65d2fb26fc8d7d80ed4cb07dfe88655891eaefa842fd91afc24/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/dc4ba310eaa3457da757fdcb0bd9364d9adeb224f3cec1b2312eb943dc155d47/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/fffc3ffdbab815637013987b591922b673bd7b5f6e2186c515e4d153eb73f0d3/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/405012bdbfd9b5c7c318b576758b0f622fe2eef51bc1dc31c6ff8998a55298bf/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/7ed14306b37ef9bd182f0a57fa8854bbd42e7e26ba64182fb0bc37ab6a8d2568/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/ca73a4831b41c5fc02bb7b291a6d49f83cddfd05864a82de3d14c239bdc99c30/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/e71b2e0501a4b78f39c1e372831762171ea5dc002e76d114c2430feace4b78e1/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/bdf9b9628e39dd2e3ab979b3c107d88906f7ef4d299343f4125639ac000a85ce/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/b3016baa2f979352ea5f8913356b618474fbc81d9418bd523885eda615ffa0cf/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/875df11199df5da1be6699d8be3627e2191ad48c343dd10952a0b73b693da335/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/6dfb69a1c22c9778995c0b5023efdc71d4f8f1ae90528b685bd5885adff87ad8/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/e57db5c8ab86bbe9374547c2131db442fee4bd292853889eb7c6e66b1f4c13a8/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/30aa87e9dda8d09547120b3a5d62692cef11c96e32cc8434c6632bb1988dd077/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/716015bfadcd7f9dcda867366a4c76757ee29a522b72b6b8bdb053766d9f4d82/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/4cea3bfc5b863b7eaa41659a3746c923ccb7d7b6b962c6c1b44a115c824b7478/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/61d49d12a94b4db016e354808cc527c07cef93dd91f7969cf5675daa471aedc7/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/0ea45f0dfc412cf211c64857e370bde8d0d1fbe57247d1136ce18e223e3c1972/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-399:{mountpoint:/var/lib/containers/storage/overlay/ab5d2a947d5359125dbe158fdb7c94d25b3266698dfc5a625c49fdf6049a6683/merged major:0 minor:399 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/dd393366fdf97d8463e3794363296ace311b8e75068d0efd4af2096c5b0ec0c5/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/08912590b2a1c0a48ff5e38fb476f989b236c327470b0c30452a3bbb77950058/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/d04eb27ee1141fe13f4b161dac50bfbd0834a9ed51dcae8c8d2e0cbbeeb0aab2/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/d1023f915b009aef8a497585a0d448e82990553aa014e63fd5f0ed4abf89941a/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/9dd3b0e344721c0ee81baf3bba9425e47c2026669f7feb7d96b2b3fc5f9c62c8/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/bbd61094f31f0b493b2c4bd8de8690237b9921bfc92f0b219b4aedb93ee7b4db/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-453:{mountpoint:/var/lib/containers/storage/overlay/7b84a78f74c99a3b72e0725cf34d3e18907c036fc5e6f3bbea72e927d3e4406e/merged major:0 minor:453 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/46368bdaebdde91086c733e23b07c3773c3a706073ea921fa1200f253f335a9b/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/0f6e093fe5600e940a26f50d74d52abec589d901ef689ae8266a9d7c081219cb/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-463:{mountpoint:/var/lib/containers/storage/overlay/f0d7900e56e11a100e462eb2747e747ef727587632b8c3e044fca78669acb67e/merged major:0 minor:463 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/09c13c3368ef285b0019acd43503bce6c39e94188c849f6cffe32e631dc4788d/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/cbd8aa059e4c668699bd99bcc1c82a639b0be943ea5b0062e83d03cf8ab97522/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/a1b95e9e4a21281bf98fa6bed527062dca36df491d7094cf29b3eef8f10b9669/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/692cc46063c76fe33e329f271375d1713768ff2eeb8b8ac6ad6d372d5110e2f8/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-497:{mountpoint:/var/lib/containers/storage/overlay/12729f790244001c47948eec080d3e232d694fe7f1d85e98b4e34624cb810db4/merged major:0 minor:497 fsType:overlay blockSize:0} overlay_0-499:{mountpoint:/var/lib/containers/storage/overlay/fcf3e6cfa81545df745d9aea76fc6d0cf391a85901c519a7021f878f130faa2e/merged major:0 minor:499 fsType:overlay blockSize:0} overlay_0-501:{mountpoint:/var/lib/containers/storage/overlay/21639ec32935c1c4171beff350df6e4557b03c7ab2fb9ebdce768353dc67249d/merged major:0 minor:501 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/143479c8cde6f05ae7329b036addd82e8cdf36983c2c1c1859c795685d7016f7/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-509:{mountpoint:/var/lib/containers/storage/overlay/a63be1519b421848adb8799eb203a39b3172eb91be675f9c10c35ba314a5f3b9/merged major:0 minor:509 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/d3101e8497d13f1bb02fa0f6da5eca1965df907b06d23b4abc8a10d1eb2a36dc/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-514:{mountpoint:/var/lib/containers/storage/overlay/a50f74a59bd04688901bee8639cb28e0d63f39386563934c90da2098f527ac50/merged major:0 minor:514 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f41585b9d24e869cb9e6e0858ffb9061923e60ffcf7c5e632a3fad34783a18ea/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ae948c188afd6a6923adc33630548898b78ea15ea137ce6c9fb89ef17aee98bd/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/4296a9166fd63197ba243567b142fb222a0333787a2b6aa648bdfb130e603940/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/594976b9d8ef94ba4e3053accc694d03033be10ea83dfe5fbb27ec1d1f5aca0c/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c268260bd87c8d3d95e78cffd380e89bea5f9ac221b4d0c41c2f4f5bdd0cd627/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/48120ed9099fcf133d56b14f2709743edf49d5a44d9651d909dd4d96f1017df3/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/4fae4492062cf9ea47720143d8917c096633cb68f649c579c030cffb9ab64074/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/38a676be7b04557fd34834a39e9889b37f946e8266445a1ceab67ddd753829e4/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/f14a413e5f47032963cdab6203ab18cf0087d7de4401acb33174b3ac085d93a0/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/e74c5e6b090329765320c33a29d0bdc4e06a610da60a8cdc60f2db3d524570fb/merged major:0 minor:98 fsType:overlay blockSize:0}]
Oct 30 12:45:58.476772 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.475879    5893 manager.go:217] Machine: {Timestamp:2026-10-30 12:45:58.475418739 +0000 UTC m=+0.058848422 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3122699 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a6b771ad6894acb540105234eb7f SystemUUID:ec28a6b7-71ad-6894-acb5-40105234eb7f BootID:077beaf1-19b2-4414-bc61-ddcbbe9d8b7d Filesystems:[{Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-453 DeviceMajor:0 DeviceMinor:453 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/afb494b7-75a0-400e-be16-74d8d6882460/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:273 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:448 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-497 DeviceMajor:0 DeviceMinor:497 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/52f5c091-40a2-46e8-9214-ddf02da33d0b/volumes/kubernetes.io~projected/kube-api-access-86mph DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/afb494b7-75a0-400e-be16-74d8d6882460/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~projected/kube-api-access-cljmm DeviceMajor:0 DeviceMinor:281 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:272 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6fdc177b4754e39d1a1dabb5ddb43e599c64550858fca406030492b3a2072321/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4eab01a9fa445d90b530a1373c3ffa1610fa98e01cba407c987b00827a6edc17/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/89288ada-24c5-46af-ad86-7d21935ca939/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/afb494b7-75a0-400e-be16-74d8d6882460/volumes/kubernetes.io~projected/kube-api-access-9x2pz DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86db01e788f2fc0e7e4e20864659d62fbe73c296d6217476e2cbb5065b738b3a/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f6897566-da42-4b8d-ba85-7ec304624f2b/volumes/kubernetes.io~projected/kube-api-access-dh9mp DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:443 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b17184b-71db-4557-ae32-062ae53a9d53/volumes/kubernetes.io~projected/kube-api-access-b4zvz DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:270 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba6439a7-7042-4edf-aa9c-6d33b168873d/volumes/kubernetes.io~projected/kube-api-access-skwdd DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eebaba938b60e759e94218534a845423109b6af2546f7607e8f86f205420d311/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:304 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/963f75f5-2539-456a-ae78-267d5b002bab/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:347 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:446 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:447 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5868626f089311f71683d77ca857047ffca6c6148d29170c11553c58803f23b1/userdata/shm DeviceMajor:0 DeviceMinor:493 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-509 DeviceMajor:0 DeviceMinor:509 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c8c3435-b6de-4aea-b596-970c9e17985c/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:310 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/35840c58-b7a6-438e-aed5-6c0d9764a793/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:351 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:444 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80a9b43884d5c96da6c6eafdf9963cd816b6a79de6a12ec8eb1b7cf364f5ab63/userdata/shm DeviceMajor:0 DeviceMinor:451 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/12dc6cc351023a89b3b2c74b20fb99e2cf2e49a084ed619bb6247ff9a2fdc8c0/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f6897566-da42-4b8d-ba85-7ec304624f2b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:309 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:271 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dccd197b-ec88-445b-8de9-ff83832112bd/volumes/kubernetes.io~projected/kube-api-access-rlm6t DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89288ada-24c5-46af-ad86-7d21935ca939/volumes/kubernetes.io~projected/kube-api-access-mpcj9 DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/416452e21dafcd020d6cce7283ba268ad5811bcfa51e211b076baafe482a1e6e/userdata/shm DeviceMajor:0 DeviceMinor:265 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/429e3a90-1868-4632-8631-5c0207978063/volumes/kubernetes.io~projected/kube-api-access-wm2ml DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/963f75f5-2539-456a-ae78-267d5b002bab/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:341 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7824adac02dc9da6019a262bbfff2cae0b9176774c857f6edd152adbd477223b/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:441 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:442 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:312 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:483 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:308 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8704082ecaafee1761693042fddc9c101ad08d9f08f9e9af632856b129dfcc58/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:274 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:275 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02cd5d46eed573ed4f4904ba91544ea05df502d39040ed3ff2df9a21f6043c72/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:311 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:275 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/963f75f5-2539-456a-ae78-267d5b002bab/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:346 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/963f75f5-2539-456a-ae78-267d5b002bab/volumes/kubernetes.io~projected/kube-api-access-8w7lh DeviceMajor:0 DeviceMinor:348 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:275 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:432 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e/volumes/kubernetes.io~projected/kube-api-access-lcxs7 DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9b17184b-71db-4557-ae32-062ae53a9d53/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:443 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/34fc63c2-b395-4702-a2e4-134161c0d50c/volumes/kubernetes.io~projected/kube-api-access-gq4cv DeviceMajor:0 DeviceMinor:488 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f6897566-da42-4b8d-ba85-7ec304624f2b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ba6439a7-7042-4edf-aa9c-6d33b168873d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/volumes/kubernetes.io~projected/kube-api-access-sfvck DeviceMajor:0 DeviceMinor:313 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-501 DeviceMajor:0 DeviceMinor:501 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c8c3435-b6de-4aea-b596-970c9e17985c/volumes/kubernetes.io~projected/kube-api-access-rkwhq DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3c29c3db2ac1fc0c5bda0cdd5d028040453bfd2da80822c25c6946f7cf4f3bd/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d3d2dc6f41867ef363f9445d7796544e423b55896729f29a7eb077f8bc2f3c7/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79d36201-724a-4cea-be2b-2e23dbf45682/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:261 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:268 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba6439a7-7042-4edf-aa9c-6d33b168873d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79d36201-724a-4cea-be2b-2e23dbf45682/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:255 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:271 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bef9665aa0a5c505102fa3df13ce1b0e6f5acff3fea328a836dc20a91e4f4525/userdata/shm DeviceMajor:0 DeviceMinor:349 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:275 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-399 DeviceMajor:0 DeviceMinor:399 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/36c3843d-81b7-4554-80ab-f6fc5901f488/volumes/kubernetes.io~projected/kube-api-access-9whn4 DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/408c5c75-c039-4dc1-89c7-6b0f4412aa59/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:487 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81359d9d6eec84462ae650df8ff4d12d53d7e7ab668d7e0d50edd4b850960e4b/userdata/shm DeviceMajor:0 DeviceMinor:491 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9b17184b-71db-4557-ae32-062ae53a9d53/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:276 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18099047513f5ac201e212b67ae436d92c39eda836ffe1c9bebab71544c5cd31/userdata/shm DeviceMajor:0 DeviceMinor:320 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:449 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:443 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8053a33-e1bf-43f4-b23b-34694e7ce042/volumes/kubernetes.io~projected/kube-api-access-sm9qn DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b0eab2da-9812-424c-9be0-af423308d43c/volumes/kubernetes.io~projected/kube-api-access-sllfq DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/89288ada-24c5-46af-ad86-7d21935ca939/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:431 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:445 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-514 DeviceMajor:0 DeviceMinor:514 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes/kubernetes.io~projected/kube-api-access-789l8 DeviceMajor:0 DeviceMinor:450 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-463 DeviceMajor:0 DeviceMinor:463 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/9bbdf052-fa04-4e1c-983a-04b313a0cdfb/volumes/kubernetes.io~projected/kube-api-access-x8s98 DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02210ac7103cb8fee0c9d6922b3f774b7409cf3c3e38509722954e36eb165f47/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce9500fc0e9de19d8eab6cfa84c7dcdf6aa309e4b785f46c662933584e330b50/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265/volumes/kubernetes.io~projected/kube-api-access-6vkrz DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/35840c58-b7a6-438e-aed5-6c0d9764a793/volumes/kubernetes.io~projected/kube-api-access-b87bm DeviceMajor:0 DeviceMinor:352 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2688554654a541a3f1221350f62a06f8fdd4a03976f114be264eee10b407512f/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:443 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-499 DeviceMajor:0 DeviceMinor:499 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b0eab2da-9812-424c-9be0-af423308d43c/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:269 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93bd4adc1974ea9c1a5ddfb97845c697f9be6f6b2f38715a630afc2f75214529/userdata/shm DeviceMajor:0 DeviceMinor:267 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a1bdce8d58370fdadfaf068b0920e59bb20f3e3f7d66e53d3a6646e9a2d8753e/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/152eef17a3f22f68eb64380aa1414873f87aa636e72173280703463843e98214/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a5a551bf44c18777ce165235fae8247aea1e9c746b181bc88d8b0b43821c2d4/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/79d36201-724a-4cea-be2b-2e23dbf45682/volumes/kubernetes.io~projected/kube-api-access-c6rml DeviceMajor:0 DeviceMinor:264 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49cf0d1aa23904c95d375634466822ecf2976144027b07593e3f1b1cefd43448/userdata/shm DeviceMajor:0 DeviceMinor:489 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5d6e88047a69ac274c0d0dca01c89c9262f9729b1637e7fd35832b473852484d/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:18099047513f5ac MacAddress:d6:db:af:c7:27:57 Speed:10000 Mtu:8901} {Name:2688554654a541a MacAddress:e2:b7:f2:9c:0a:30 Speed:10000 Mtu:8901} {Name:416452e21dafcd0 MacAddress:2a:68:7b:df:34:aa Speed:10000 Mtu:8901} {Name:49cf0d1aa23904c MacAddress:a2:e1:b6:6c:67:9f Speed:10000 Mtu:8901} {Name:4eab01a9fa445d9 MacAddress:8a:fe:7e:02:58:93 Speed:10000 Mtu:8901} {Name:5868626f089311f MacAddress:aa:69:80:21:c7:36 Speed:10000 Mtu:8901} {Name:5d6e88047a69ac2 MacAddress:62:9b:f7:c3:2b:8c Speed:10000 Mtu:8901} {Name:7824adac02dc9da MacAddress:62:96:9e:a0:c2:a0 Speed:10000 Mtu:8901} {Name:80a9b43884d5c96 MacAddress:92:97:74:d7:66:1e Speed:10000 Mtu:8901} {Name:81359d9d6eec844 MacAddress:1e:15:83:8c:8e:b8 Speed:10000 Mtu:8901} {Name:93bd4adc1974ea9 MacAddress:9e:7f:d2:29:41:09 Speed:10000 Mtu:8901} {Name:9a5a551bf44c187 MacAddress:7a:61:cd:1d:3d:89 Speed:10000 Mtu:8901} {Name:b3c29c3db2ac1fc MacAddress:ae:75:61:7d:f7:fa Speed:10000 Mtu:8901} {Name:bef9665aa0a5c50 MacAddress:d6:5b:02:19:bf:c0 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:db:92:f4:eb Speed:0 Mtu:9001} {Name:br-int MacAddress:a2:6f:ee:ca:ee:89 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:db:92:f4:eb Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:e6:35:98:8f:0c:3e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2a:4f:bc:6c:e1:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 12:45:58.476772 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.476344    5893 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 12:45:58.476772 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.476386    5893 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 12:45:58.476772 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.476679    5893 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 12:45:58.477070 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.476701    5893 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-232.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 12:45:58.477070 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.476819    5893 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 12:45:58.477070 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.476827    5893 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 12:45:58.477070 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.476840    5893 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:45:58.477070 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.476853    5893 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 12:45:58.477070 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.476921    5893 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 12:45:58.477244 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.477080    5893 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 12:45:58.477244 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.477190    5893 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 12:45:58.477244 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.477222    5893 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 12:45:58.477244 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.477231    5893 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 12:45:58.477244 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.477242    5893 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 12:45:58.477410 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.477251    5893 kubelet.go:405] "Adding apiserver pod source"
Oct 30 12:45:58.477410 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.477259    5893 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 12:45:58.479113 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.478913    5893 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 12:45:58.479113 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.478934    5893 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:45:58.479577 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.479552    5893 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 12:45:58.482197 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482178    5893 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 12:45:58.482197 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482194    5893 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 12:45:58.482448 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482339    5893 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 12:45:58.482599 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482583    5893 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 12:45:58.482655 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482625    5893 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 12:45:58.482749 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482738    5893 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 12:45:58.482804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482758    5893 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 12:45:58.482804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482768    5893 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 12:45:58.482804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482777    5893 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 12:45:58.482804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482785    5893 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 12:45:58.482804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482796    5893 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 12:45:58.483036 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482806    5893 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 12:45:58.483036 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482815    5893 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 12:45:58.483036 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482826    5893 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 12:45:58.483036 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482837    5893 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 12:45:58.483036 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482861    5893 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 12:45:58.483036 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482875    5893 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 12:45:58.483036 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482915    5893 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 12:45:58.483036 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.482924    5893 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 12:45:58.486392 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.486377    5893 server.go:1263] "Started kubelet"
Oct 30 12:45:58.486791 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.486439    5893 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 12:45:58.486791 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.486485    5893 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 12:45:58.486791 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.486559    5893 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 12:45:58.486923 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.486862    5893 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 12:45:58.486925 ip-10-0-141-232 systemd[1]: Started Kubernetes Kubelet.
Oct 30 12:45:58.487951 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.487920    5893 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 12:45:58.492893 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.492559    5893 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 12:45:58.492893 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.492580    5893 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 12:45:58.492893 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.492626    5893 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:39:49 +0000 UTC" deadline="2028-04-12 07:47:28.473787282 +0000 UTC"
Oct 30 12:45:58.492893 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.492648    5893 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12715h1m29.981143649s"
Oct 30 12:45:58.492893 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.492673    5893 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 12:45:58.492893 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.492683    5893 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 12:45:58.492893 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:58.492721    5893 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-232.ec2.internal\" not found"
Oct 30 12:45:58.492893 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.492771    5893 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 12:45:58.493904 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.493825    5893 factory.go:55] Registering systemd factory
Oct 30 12:45:58.493904 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.493845    5893 factory.go:223] Registration of the systemd container factory successfully
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.494650    5893 factory.go:153] Registering CRI-O factory
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.494670    5893 factory.go:223] Registration of the crio container factory successfully
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.494725    5893 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.494744    5893 factory.go:103] Registering Raw factory
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.494760    5893 manager.go:1199] Started watching for new ooms in manager
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.495072    5893 manager.go:319] Starting recovery of all containers
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.495868    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afb494b7-75a0-400e-be16-74d8d6882460" volumeName="kubernetes.io/configmap/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.495903    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afb494b7-75a0-400e-be16-74d8d6882460" volumeName="kubernetes.io/configmap/afb494b7-75a0-400e-be16-74d8d6882460-metrics-client-ca" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.495917    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6897566-da42-4b8d-ba85-7ec304624f2b" volumeName="kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-env-overrides" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.495929    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c8c3435-b6de-4aea-b596-970c9e17985c" volumeName="kubernetes.io/projected/6c8c3435-b6de-4aea-b596-970c9e17985c-kube-api-access-rkwhq" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.495949    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89288ada-24c5-46af-ad86-7d21935ca939" volumeName="kubernetes.io/secret/89288ada-24c5-46af-ad86-7d21935ca939-openshift-state-metrics-tls" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.495963    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8053a33-e1bf-43f4-b23b-34694e7ce042" volumeName="kubernetes.io/projected/a8053a33-e1bf-43f4-b23b-34694e7ce042-kube-api-access-sm9qn" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.495985    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-thanos-prometheus-http-client-file" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.495998    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb5dfe9c-5cf0-43af-875c-3bc0fd07c265" volumeName="kubernetes.io/empty-dir/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-data-volume" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496013    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4edeee-f5a5-497d-a53c-2a6c82a3fe85" volumeName="kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496026    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79d36201-724a-4cea-be2b-2e23dbf45682" volumeName="kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496040    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-metrics-client-certs" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496053    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4edeee-f5a5-497d-a53c-2a6c82a3fe85" volumeName="kubernetes.io/projected/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-kube-api-access-sfvck" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496064    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63c94b3e-9026-4fba-9788-03f2d981b53b" volumeName="kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-config-out" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496076    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79d36201-724a-4cea-be2b-2e23dbf45682" volumeName="kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496093    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="963f75f5-2539-456a-ae78-267d5b002bab" volumeName="kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-secret-metrics-server-tls" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496104    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b17184b-71db-4557-ae32-062ae53a9d53" volumeName="kubernetes.io/secret/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496116    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d08a5487-7eaa-4438-8fcd-3ead8dc6c20e" volumeName="kubernetes.io/secret/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e-agent-certs" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496131    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-metrics-client-ca" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496143    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccd197b-ec88-445b-8de9-ff83832112bd" volumeName="kubernetes.io/projected/dccd197b-ec88-445b-8de9-ff83832112bd-kube-api-access-rlm6t" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496155    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="34fc63c2-b395-4702-a2e4-134161c0d50c" volumeName="kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496170    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79d36201-724a-4cea-be2b-2e23dbf45682" volumeName="kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496181    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="963f75f5-2539-456a-ae78-267d5b002bab" volumeName="kubernetes.io/configmap/963f75f5-2539-456a-ae78-267d5b002bab-metrics-server-audit-profiles" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496221    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-db" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496234    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="408c5c75-c039-4dc1-89c7-6b0f4412aa59" volumeName="kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496244    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c8c3435-b6de-4aea-b596-970c9e17985c" volumeName="kubernetes.io/configmap/6c8c3435-b6de-4aea-b596-970c9e17985c-config-volume" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496255    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b17184b-71db-4557-ae32-062ae53a9d53" volumeName="kubernetes.io/projected/9b17184b-71db-4557-ae32-062ae53a9d53-kube-api-access-b4zvz" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496267    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-2" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496306    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4edeee-f5a5-497d-a53c-2a6c82a3fe85" volumeName="kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-grpc-tls" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496323    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="429e3a90-1868-4632-8631-5c0207978063" volumeName="kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496337    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bbdf052-fa04-4e1c-983a-04b313a0cdfb" volumeName="kubernetes.io/configmap/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-cni-binary-copy" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496366    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6897566-da42-4b8d-ba85-7ec304624f2b" volumeName="kubernetes.io/projected/f6897566-da42-4b8d-ba85-7ec304624f2b-kube-api-access-dh9mp" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496389    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afb494b7-75a0-400e-be16-74d8d6882460" volumeName="kubernetes.io/projected/afb494b7-75a0-400e-be16-74d8d6882460-kube-api-access-9x2pz" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496401    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="429e3a90-1868-4632-8631-5c0207978063" volumeName="kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-cni-binary-copy" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496415    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52f5c091-40a2-46e8-9214-ddf02da33d0b" volumeName="kubernetes.io/projected/52f5c091-40a2-46e8-9214-ddf02da33d0b-kube-api-access-86mph" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496426    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afb494b7-75a0-400e-be16-74d8d6882460" volumeName="kubernetes.io/empty-dir/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-textfile" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496439    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-tls-assets" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496451    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="429e3a90-1868-4632-8631-5c0207978063" volumeName="kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496463    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-1" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496475    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-config-out" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496487    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-tls" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496499    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6897566-da42-4b8d-ba85-7ec304624f2b" volumeName="kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496513    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4edeee-f5a5-497d-a53c-2a6c82a3fe85" volumeName="kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496525    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e" volumeName="kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496537    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b17184b-71db-4557-ae32-062ae53a9d53" volumeName="kubernetes.io/secret/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-tls" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496550    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-serving-certs-ca-bundle" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496563    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6897566-da42-4b8d-ba85-7ec304624f2b" volumeName="kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-ovnkube-config" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496575    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6897566-da42-4b8d-ba85-7ec304624f2b" volumeName="kubernetes.io/secret/f6897566-da42-4b8d-ba85-7ec304624f2b-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496589    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63c94b3e-9026-4fba-9788-03f2d981b53b" volumeName="kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496601    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bbdf052-fa04-4e1c-983a-04b313a0cdfb" volumeName="kubernetes.io/configmap/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-daemon-config" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496615    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dccd197b-ec88-445b-8de9-ff83832112bd" volumeName="kubernetes.io/configmap/dccd197b-ec88-445b-8de9-ff83832112bd-serviceca" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496626    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35840c58-b7a6-438e-aed5-6c0d9764a793" volumeName="kubernetes.io/projected/35840c58-b7a6-438e-aed5-6c0d9764a793-kube-api-access-b87bm" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496638    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63c94b3e-9026-4fba-9788-03f2d981b53b" volumeName="kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-kube-api-access-cljmm" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496653    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63c94b3e-9026-4fba-9788-03f2d981b53b" volumeName="kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-config-volume" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496665    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c8c3435-b6de-4aea-b596-970c9e17985c" volumeName="kubernetes.io/empty-dir/6c8c3435-b6de-4aea-b596-970c9e17985c-tmp-dir" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496683    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496697    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="825a2efb-6a8c-4bdd-bf89-cca7ee710f8e" volumeName="kubernetes.io/projected/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-kube-api-access-lcxs7" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496717    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63c94b3e-9026-4fba-9788-03f2d981b53b" volumeName="kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-tls-assets" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496729    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63c94b3e-9026-4fba-9788-03f2d981b53b" volumeName="kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-main-tls" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496741    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63c94b3e-9026-4fba-9788-03f2d981b53b" volumeName="kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496755    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79d36201-724a-4cea-be2b-2e23dbf45682" volumeName="kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496777    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afb494b7-75a0-400e-be16-74d8d6882460" volumeName="kubernetes.io/secret/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-tls" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496791    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0eab2da-9812-424c-9be0-af423308d43c" volumeName="kubernetes.io/secret/b0eab2da-9812-424c-9be0-af423308d43c-cert" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496802    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-config" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496814    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4edeee-f5a5-497d-a53c-2a6c82a3fe85" volumeName="kubernetes.io/empty-dir/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-tmp" seLinuxMountContext=""
Oct 30 12:45:58.496775 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496825    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63c94b3e-9026-4fba-9788-03f2d981b53b" volumeName="kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496838    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63c94b3e-9026-4fba-9788-03f2d981b53b" volumeName="kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496849    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89288ada-24c5-46af-ad86-7d21935ca939" volumeName="kubernetes.io/configmap/89288ada-24c5-46af-ad86-7d21935ca939-metrics-client-ca" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496862    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89288ada-24c5-46af-ad86-7d21935ca939" volumeName="kubernetes.io/secret/89288ada-24c5-46af-ad86-7d21935ca939-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496875    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79d36201-724a-4cea-be2b-2e23dbf45682" volumeName="kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496886    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba6439a7-7042-4edf-aa9c-6d33b168873d" volumeName="kubernetes.io/projected/ba6439a7-7042-4edf-aa9c-6d33b168873d-kube-api-access-skwdd" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496896    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-grpc-tls" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496908    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb5dfe9c-5cf0-43af-875c-3bc0fd07c265" volumeName="kubernetes.io/projected/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-kube-api-access-6vkrz" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496919    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36c3843d-81b7-4554-80ab-f6fc5901f488" volumeName="kubernetes.io/projected/36c3843d-81b7-4554-80ab-f6fc5901f488-kube-api-access-9whn4" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496932    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="963f75f5-2539-456a-ae78-267d5b002bab" volumeName="kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-secret-metrics-server-client-certs" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496956    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b17184b-71db-4557-ae32-062ae53a9d53" volumeName="kubernetes.io/empty-dir/9b17184b-71db-4557-ae32-062ae53a9d53-volume-directive-shadow" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496970    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496982    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb5dfe9c-5cf0-43af-875c-3bc0fd07c265" volumeName="kubernetes.io/secret/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.496996    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4edeee-f5a5-497d-a53c-2a6c82a3fe85" volumeName="kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-tls" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497007    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="963f75f5-2539-456a-ae78-267d5b002bab" volumeName="kubernetes.io/empty-dir/963f75f5-2539-456a-ae78-267d5b002bab-audit-log" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497019    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79d36201-724a-4cea-be2b-2e23dbf45682" volumeName="kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497031    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b17184b-71db-4557-ae32-062ae53a9d53" volumeName="kubernetes.io/configmap/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497043    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b17184b-71db-4557-ae32-062ae53a9d53" volumeName="kubernetes.io/configmap/9b17184b-71db-4557-ae32-062ae53a9d53-metrics-client-ca" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497054    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8053a33-e1bf-43f4-b23b-34694e7ce042" volumeName="kubernetes.io/configmap/a8053a33-e1bf-43f4-b23b-34694e7ce042-iptables-alerter-script" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497067    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497081    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35840c58-b7a6-438e-aed5-6c0d9764a793" volumeName="kubernetes.io/secret/35840c58-b7a6-438e-aed5-6c0d9764a793-monitoring-plugin-cert" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497092    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c8c3435-b6de-4aea-b596-970c9e17985c" volumeName="kubernetes.io/secret/6c8c3435-b6de-4aea-b596-970c9e17985c-metrics-tls" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497103    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79d36201-724a-4cea-be2b-2e23dbf45682" volumeName="kubernetes.io/empty-dir/79d36201-724a-4cea-be2b-2e23dbf45682-tmp" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497114    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="afb494b7-75a0-400e-be16-74d8d6882460" volumeName="kubernetes.io/secret/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497126    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba6439a7-7042-4edf-aa9c-6d33b168873d" volumeName="kubernetes.io/empty-dir/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-tuned" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497139    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="52f5c091-40a2-46e8-9214-ddf02da33d0b" volumeName="kubernetes.io/empty-dir/52f5c091-40a2-46e8-9214-ddf02da33d0b-tmp-dir" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497151    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba6439a7-7042-4edf-aa9c-6d33b168873d" volumeName="kubernetes.io/empty-dir/ba6439a7-7042-4edf-aa9c-6d33b168873d-tmp" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497163    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63c94b3e-9026-4fba-9788-03f2d981b53b" volumeName="kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-main-db" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497174    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb5dfe9c-5cf0-43af-875c-3bc0fd07c265" volumeName="kubernetes.io/configmap/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497185    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="963f75f5-2539-456a-ae78-267d5b002bab" volumeName="kubernetes.io/configmap/963f75f5-2539-456a-ae78-267d5b002bab-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497199    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bbdf052-fa04-4e1c-983a-04b313a0cdfb" volumeName="kubernetes.io/projected/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-kube-api-access-x8s98" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497210    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4edeee-f5a5-497d-a53c-2a6c82a3fe85" volumeName="kubernetes.io/configmap/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-metrics-client-ca" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497221    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63c94b3e-9026-4fba-9788-03f2d981b53b" volumeName="kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-web-config" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497234    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79d36201-724a-4cea-be2b-2e23dbf45682" volumeName="kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497246    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="963f75f5-2539-456a-ae78-267d5b002bab" volumeName="kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-client-ca-bundle" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497256    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-0" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497296    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-kube-api-access-789l8" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497311    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497323    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4edeee-f5a5-497d-a53c-2a6c82a3fe85" volumeName="kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497335    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0eab2da-9812-424c-9be0-af423308d43c" volumeName="kubernetes.io/projected/b0eab2da-9812-424c-9be0-af423308d43c-kube-api-access-sllfq" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497345    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d4edeee-f5a5-497d-a53c-2a6c82a3fe85" volumeName="kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497356    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63c94b3e-9026-4fba-9788-03f2d981b53b" volumeName="kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-metrics-client-ca" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497366    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d08a5487-7eaa-4438-8fcd-3ead8dc6c20e" volumeName="kubernetes.io/configmap/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e-konnectivity-ca" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497403    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-web-config" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497415    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="429e3a90-1868-4632-8631-5c0207978063" volumeName="kubernetes.io/projected/429e3a90-1868-4632-8631-5c0207978063-kube-api-access-wm2ml" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497427    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89288ada-24c5-46af-ad86-7d21935ca939" volumeName="kubernetes.io/projected/89288ada-24c5-46af-ad86-7d21935ca939-kube-api-access-mpcj9" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497438    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d457ae45-4f12-43bf-b6a5-634a29981a55" volumeName="kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-kube-rbac-proxy" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497456    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="63c94b3e-9026-4fba-9788-03f2d981b53b" volumeName="kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-cluster-tls-config" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497486    5893 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="963f75f5-2539-456a-ae78-267d5b002bab" volumeName="kubernetes.io/projected/963f75f5-2539-456a-ae78-267d5b002bab-kube-api-access-8w7lh" seLinuxMountContext=""
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497499    5893 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 12:45:58.500017 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.497507    5893 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 12:45:58.520539 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.520395    5893 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 12:45:58.520741 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.520722    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 12:45:58.521949 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.521924    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 12:45:58.522980 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.522957    5893 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 12:45:58.522980 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.522981    5893 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 12:45:58.523174 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.523002    5893 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 12:45:58.523243 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:58.523211    5893 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 12:45:58.539692 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.539669    5893 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:45:58.539815 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.539802    5893 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 12:45:58.542009 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.541977    5893 generic.go:356] "Generic (PLEG): container finished" podID="e8bd9c92b47bd47faf7dc12d1bdd0a47" containerID="4aeda4a396f0aee7a6eb95eea247d68b20c3fadbfda23e45bf5d714dfc1def15" exitCode=0
Oct 30 12:45:58.542447 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.542417    5893 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:45:58.550419 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.550391    5893 generic.go:356] "Generic (PLEG): container finished" podID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerID="09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6" exitCode=0
Oct 30 12:45:58.556899 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.556874    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 12:45:58.557436 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.557406    5893 generic.go:356] "Generic (PLEG): container finished" podID="f6897566-da42-4b8d-ba85-7ec304624f2b" containerID="b9a72fd6984e27eaa724167e0ae42f05f2f36717495836ab9e6c2409c87d4656" exitCode=1
Oct 30 12:45:58.560445 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.560423    5893 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:45:58.560710 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.560691    5893 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 12:45:58.564862 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.564840    5893 generic.go:356] "Generic (PLEG): container finished" podID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerID="af2b81871b51f98092e2ca59b7ad6cd7e58f4d8f62a86ba6b153408ca3a82817" exitCode=0
Oct 30 12:45:58.568902 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.568874    5893 generic.go:356] "Generic (PLEG): container finished" podID="afb494b7-75a0-400e-be16-74d8d6882460" containerID="a03252c3e03967e9d31311987ac868d680e7f56ff8bacaba7d67166ca53ff5eb" exitCode=0
Oct 30 12:45:58.588124 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.588093    5893 generic.go:356] "Generic (PLEG): container finished" podID="429e3a90-1868-4632-8631-5c0207978063" containerID="01e224997aa4ad2c8ccf08ac6523e3c76f4d73087390a72f2ce9eace4539f18b" exitCode=0
Oct 30 12:45:58.588124 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.588126    5893 generic.go:356] "Generic (PLEG): container finished" podID="429e3a90-1868-4632-8631-5c0207978063" containerID="48a7c7cae6da64cdba507f18d16fd403026286637b4cda7b1555829d695ad085" exitCode=0
Oct 30 12:45:58.588292 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.588139    5893 generic.go:356] "Generic (PLEG): container finished" podID="429e3a90-1868-4632-8631-5c0207978063" containerID="91f48fe4a7a3c66734b6627edbcbcf817cd532eaa58a614c4cf82ab80d758597" exitCode=0
Oct 30 12:45:58.588292 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.588151    5893 generic.go:356] "Generic (PLEG): container finished" podID="429e3a90-1868-4632-8631-5c0207978063" containerID="dadf45da29b49d09436574031b6626e88ffafe3094539a475029821fee8ce8c1" exitCode=0
Oct 30 12:45:58.588292 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.588156    5893 generic.go:356] "Generic (PLEG): container finished" podID="429e3a90-1868-4632-8631-5c0207978063" containerID="02fd14a51f8d2c34d09b1a0ca6389849547c39625ef73fb76a00bb6955d1e7ca" exitCode=0
Oct 30 12:45:58.588292 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.588161    5893 generic.go:356] "Generic (PLEG): container finished" podID="429e3a90-1868-4632-8631-5c0207978063" containerID="f2ddbdc2aee542040e3b54d20e69e2344a35139c6067622d326fda789888ae0b" exitCode=0
Oct 30 12:45:58.615316 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.615243    5893 manager.go:324] Recovery completed
Oct 30 12:45:58.623411 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:58.623394    5893 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 12:45:58.641620 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.641600    5893 cpu_manager.go:226] "Starting" policy="none"
Oct 30 12:45:58.641620 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.641614    5893 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 12:45:58.641749 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.641630    5893 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 12:45:58.641779 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.641744    5893 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 12:45:58.641779 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.641763    5893 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 12:45:58.641779 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.641779    5893 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 12:45:58.641867 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.641785    5893 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 12:45:58.641867 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.641790    5893 policy_none.go:50] "Start"
Oct 30 12:45:58.641867 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.641798    5893 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 12:45:58.641867 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.641807    5893 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 12:45:58.641964 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.641930    5893 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 12:45:58.641964 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.641937    5893 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 12:45:58.641964 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.641942    5893 policy_none.go:44] "Start"
Oct 30 12:45:58.646695 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.646667    5893 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 12:45:58.650331 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.650321    5893 manager.go:341] "Starting Device Plugin manager"
Oct 30 12:45:58.650898 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:58.650345    5893 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 12:45:58.650958 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.650901    5893 server.go:87] "Starting device plugin registration server"
Oct 30 12:45:58.651152 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.651142    5893 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 12:45:58.651185 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.651153    5893 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 12:45:58.651285 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.651249    5893 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 12:45:58.651448 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.651400    5893 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 12:45:58.651523 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.651455    5893 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 12:45:58.651523 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.651462    5893 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 12:45:58.651596 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.651510    5893 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T12:45:58.651445021Z","UUID":"faf6ee40-0cc1-4ede-ac03-a953ded5cf9c","Handler":null,"Name":"","Endpoint":""}
Oct 30 12:45:58.652920 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.652901    5893 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 12:45:58.653036 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.652935    5893 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 12:45:58.653469 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:58.653442    5893 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 12:45:58.751837 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.751803    5893 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 12:45:58.753793 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.753770    5893 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 12:45:58.753896 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.753803    5893 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 12:45:58.753896 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.753813    5893 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-232.ec2.internal" event="NodeHasSufficientPID"
Oct 30 12:45:58.753896 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.753860    5893 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-232.ec2.internal"
Oct 30 12:45:58.784240 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.784216    5893 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-232.ec2.internal"
Oct 30 12:45:58.784357 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.784303    5893 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-232.ec2.internal"
Oct 30 12:45:58.824397 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.824340    5893 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"]
Oct 30 12:45:58.825016 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.825000    5893 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:58.825057 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.824998    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal" event={"ID":"e8bd9c92b47bd47faf7dc12d1bdd0a47","Type":"ContainerStarted","Data":"8b19358587860669d595e4c041a6d3d7f512d54adff3d426e48bf7972bd9bc43"}
Oct 30 12:45:58.825057 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.825039    5893 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:58.825129 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.825070    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal" event={"ID":"e8bd9c92b47bd47faf7dc12d1bdd0a47","Type":"ContainerDied","Data":"4aeda4a396f0aee7a6eb95eea247d68b20c3fadbfda23e45bf5d714dfc1def15"}
Oct 30 12:45:58.825129 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.825091    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal" event={"ID":"e8bd9c92b47bd47faf7dc12d1bdd0a47","Type":"ContainerStarted","Data":"a1bdce8d58370fdadfaf068b0920e59bb20f3e3f7d66e53d3a6646e9a2d8753e"}
Oct 30 12:45:58.825195 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.825162    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal" event={"ID":"6886c36025c26ee79576270d5200a014","Type":"ContainerStarted","Data":"09c06cb8f6ffe92d449bfd7a645ef115caecaf57fbfafd62f913842100bd71d5"}
Oct 30 12:45:58.825195 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.825178    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal" event={"ID":"6886c36025c26ee79576270d5200a014","Type":"ContainerStarted","Data":"86db01e788f2fc0e7e4e20864659d62fbe73c296d6217476e2cbb5065b738b3a"}
Oct 30 12:45:58.848662 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.848644    5893 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:45:58.848715 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:58.848683    5893 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-232.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:58.849733 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.849718    5893 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:45:58.849782 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:58.849755    5893 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:58.942179 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.942107    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8bd9c92b47bd47faf7dc12d1bdd0a47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"e8bd9c92b47bd47faf7dc12d1bdd0a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:58.942179 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.942142    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8bd9c92b47bd47faf7dc12d1bdd0a47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"e8bd9c92b47bd47faf7dc12d1bdd0a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:58.942179 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:58.942166    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6886c36025c26ee79576270d5200a014-config\") pod \"kube-apiserver-proxy-ip-10-0-141-232.ec2.internal\" (UID: \"6886c36025c26ee79576270d5200a014\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:59.042650 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.042614    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6886c36025c26ee79576270d5200a014-config\") pod \"kube-apiserver-proxy-ip-10-0-141-232.ec2.internal\" (UID: \"6886c36025c26ee79576270d5200a014\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:59.042650 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.042648    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8bd9c92b47bd47faf7dc12d1bdd0a47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"e8bd9c92b47bd47faf7dc12d1bdd0a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:59.042818 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.042664    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8bd9c92b47bd47faf7dc12d1bdd0a47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"e8bd9c92b47bd47faf7dc12d1bdd0a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:59.042818 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.042720    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6886c36025c26ee79576270d5200a014-config\") pod \"kube-apiserver-proxy-ip-10-0-141-232.ec2.internal\" (UID: \"6886c36025c26ee79576270d5200a014\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:59.042818 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.042771    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e8bd9c92b47bd47faf7dc12d1bdd0a47-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"e8bd9c92b47bd47faf7dc12d1bdd0a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:59.042818 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.042805    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e8bd9c92b47bd47faf7dc12d1bdd0a47-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" (UID: \"e8bd9c92b47bd47faf7dc12d1bdd0a47\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:59.477803 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.477766    5893 apiserver.go:52] "Watching apiserver"
Oct 30 12:45:59.523152 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.523125    5893 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 12:45:59.523803 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.523778    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-fd9xb","openshift-ingress-canary/ingress-canary-dlv29","openshift-console/console-7d587d759d-h9f5b","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal","openshift-monitoring/alertmanager-main-0","openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw","openshift-multus/network-metrics-daemon-zfp4t","openshift-network-operator/iptables-alerter-jtfpn","kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal","openshift-monitoring/prometheus-k8s-0","openshift-image-registry/node-ca-bzkw4","openshift-monitoring/metrics-server-b6fc5f989-jm9qh","openshift-monitoring/node-exporter-2xmlk","openshift-monitoring/thanos-querier-5ff77c7974-6cgw8","openshift-dns/node-resolver-tg5ns","openshift-insights/insights-runtime-extractor-k47jk","openshift-ovn-kubernetes/ovnkube-node-dg4xp","kube-system/global-pull-secret-syncer-79f2f","openshift-cluster-node-tuning-operator/tuned-gxdv2","openshift-dns/dns-default-rm7hq","openshift-monitoring/monitoring-plugin-77f4f76594-krk2q","openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t","openshift-multus/multus-additional-cni-plugins-jzgc2","openshift-network-diagnostics/network-check-target-7qcht","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv","openshift-multus/multus-x48w4"]
Oct 30 12:45:59.538032 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.538008    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 12:45:59.538626 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.538600    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 12:45:59.538769 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.538647    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 12:45:59.538769 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.538684    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 12:45:59.538877 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.538778    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 12:45:59.538877 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.538860    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-5fhhc\""
Oct 30 12:45:59.538877 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.538866    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 12:45:59.539025 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.538903    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:45:59.539025 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.538928    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mwttw\""
Oct 30 12:45:59.539025 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.538950    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 12:45:59.539025 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.539009    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 12:45:59.539213 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.539060    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 12:45:59.539213 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.539096    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 12:45:59.539213 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.539121    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6f4z2\""
Oct 30 12:45:59.539213 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.538904    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 12:45:59.539413 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.539255    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k9vxw\""
Oct 30 12:45:59.539413 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.539255    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 12:45:59.539498 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.539430    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 12:45:59.539498 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.539455    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b79n8\""
Oct 30 12:45:59.545440 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.545422    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 12:45:59.545644 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.545629    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 12:45:59.545717 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.545634    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 12:45:59.545959 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.545945    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 12:45:59.546032 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.545990    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 12:45:59.546599 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.546584    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 12:45:59.547616 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.547599    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 12:45:59.548079 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.548065    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 12:45:59.548813 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.548800    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mxwzk\""
Oct 30 12:45:59.549073 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.549055    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-trr9l\""
Oct 30 12:45:59.549349 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.549337    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w75sr\""
Oct 30 12:45:59.549397 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.549371    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6g9fh\""
Oct 30 12:45:59.549597 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.549584    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-5ldkz\""
Oct 30 12:45:59.549647 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.549586    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 12:45:59.549647 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.549600    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xlfj2\""
Oct 30 12:45:59.549647 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.549612    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 12:45:59.549647 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.549619    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g92fm\""
Oct 30 12:45:59.549859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.549845    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zngfb\""
Oct 30 12:45:59.549922 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.549912    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 12:45:59.549999 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.549987    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 12:45:59.551688 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.551674    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-l246t\""
Oct 30 12:45:59.551768 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.551726    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 12:45:59.552224 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.552208    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 12:45:59.555721 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.555704    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 12:45:59.555721 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.555709    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 12:45:59.555821 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.555788    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 12:45:59.556496 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.556479    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 12:45:59.556569 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.556493    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 12:45:59.556569 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.556497    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 12:45:59.556569 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.556497    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 12:45:59.556569 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.556539    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 12:45:59.556569 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.556547    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 12:45:59.556569 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.556552    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 12:45:59.556569 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.556549    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 12:45:59.556777 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.556577    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 12:45:59.556777 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.556581    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 12:45:59.556777 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.556596    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 12:45:59.556777 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.556625    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 12:45:59.557582 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.557566    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 12:45:59.557862 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.557850    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 12:45:59.557920 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.557885    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 12:45:59.557920 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.557898    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pglr2\""
Oct 30 12:45:59.557920 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.557902    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-njv68\""
Oct 30 12:45:59.558040 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.557930    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-svz4t\""
Oct 30 12:45:59.558040 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.557949    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 12:45:59.558040 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.557964    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 12:45:59.558040 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.557967    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 12:45:59.558146 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.558127    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 12:45:59.558200 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.558190    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 12:45:59.598349 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.596012    5893 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:59.598349 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.596552    5893 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:59.599534 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.599509    5893 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 12:45:59.617699 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.617678    5893 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:45:59.617791 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.617722    5893 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-232.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:59.618259 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.618242    5893 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 12:45:59.618369 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.618315    5893 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-232.ec2.internal"
Oct 30 12:45:59.638053 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.638027    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 12:45:59.638367 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.638351    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 12:45:59.638410 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.638387    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 12:45:59.638559 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.638544    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:45:59.638974 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.638948    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 12:45:59.638974 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.638968    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 12:45:59.639072 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.638994    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 12:45:59.639201 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639169    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 12:45:59.639201 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639180    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 12:45:59.639201 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639184    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 12:45:59.639432 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639200    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 12:45:59.639432 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639293    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 12:45:59.639432 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639333    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 12:45:59.639432 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639419    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 12:45:59.639655 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639436    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 12:45:59.639655 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639419    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 12:45:59.639655 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639571    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 12:45:59.639655 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639587    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 12:45:59.639655 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639595    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 12:45:59.639655 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639618    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 12:45:59.639655 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639625    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 12:45:59.639655 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639635    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dcr46kt5lt1rp\""
Oct 30 12:45:59.640022 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639664    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 12:45:59.640022 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639911    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 12:45:59.640022 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639929    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 12:45:59.640022 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639943    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-xx2bf\""
Oct 30 12:45:59.640022 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639943    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 12:45:59.640022 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639964    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 12:45:59.640022 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639977    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-fs22w\""
Oct 30 12:45:59.640022 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.639995    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 12:45:59.640022 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.640002    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dqrs9\""
Oct 30 12:45:59.640022 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.640006    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-s2l2p\""
Oct 30 12:45:59.640022 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.640031    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-366snq849hmfp\""
Oct 30 12:45:59.640521 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.640172    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 12:45:59.640521 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.640183    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dtr69qhp6drph\""
Oct 30 12:45:59.640521 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.640215    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gxtbq\""
Oct 30 12:45:59.640521 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.640325    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 12:45:59.640521 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.640383    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 12:45:59.640521 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.640438    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 12:45:59.640737 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.640554    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5q4kn\""
Oct 30 12:45:59.640737 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.640722    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 12:45:59.640806 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.640738    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 12:45:59.644666 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.644650    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-var-lib-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.644719 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.644682    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dccd197b-ec88-445b-8de9-ff83832112bd-serviceca\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:45:59.644719 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.644701    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89288ada-24c5-46af-ad86-7d21935ca939-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:59.644817 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.644718    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-web-config\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.644817 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.644733    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-ovnkube-script-lib\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.644817 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.644749    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-host\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.644817 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.644798    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-cni-bin\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.645001 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.644893    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.645001 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.644963    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.645099 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645014    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-web-config\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.645099 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645026    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/89288ada-24c5-46af-ad86-7d21935ca939-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:59.645099 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645081    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.645099 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645090    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-ovnkube-script-lib\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.645234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645158    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-cni-bin\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.645234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645167    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.645234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645181    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-env-overrides\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.645346 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645258    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlm6t\" (UniqueName: \"kubernetes.io/projected/dccd197b-ec88-445b-8de9-ff83832112bd-kube-api-access-rlm6t\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:45:59.645346 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645305    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-modprobe-d\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.645346 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645309    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.645346 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645333    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8s98\" (UniqueName: \"kubernetes.io/projected/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-kube-api-access-x8s98\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.645463 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645381    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.645463 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645386    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.645463 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645408    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.645463 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645432    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:59.645463 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645456    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-socket-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.645646 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645491    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-env-overrides\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.645646 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645518    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-device-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.645646 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645530    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/dccd197b-ec88-445b-8de9-ff83832112bd-serviceca\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:45:59.645646 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645544    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f5c091-40a2-46e8-9214-ddf02da33d0b-tmp-dir\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:45:59.645646 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645566    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/408c5c75-c039-4dc1-89c7-6b0f4412aa59-dbus\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:59.645646 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645592    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysctl-conf\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.645646 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645616    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-config-out\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.645646 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645639    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-run-netns\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.645865 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645680    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-wtmp\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.645865 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645679    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.645865 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645702    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52f5c091-40a2-46e8-9214-ddf02da33d0b-tmp-dir\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:45:59.645865 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645721    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-kubernetes\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.645865 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645775    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-config-out\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.645865 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645774    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm2ml\" (UniqueName: \"kubernetes.io/projected/429e3a90-1868-4632-8631-5c0207978063-kube-api-access-wm2ml\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.646064 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.645898    5893 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 12:45:59.646064 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.645916    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-config\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.646064 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.645968    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:00.145953405 +0000 UTC m=+1.729383101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-config" not registered
Oct 30 12:45:59.646064 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646002    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.646064 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646017    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.646211 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646077    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-ovn\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.646211 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646144    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-config\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.646211 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646189    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52f5c091-40a2-46e8-9214-ddf02da33d0b-hosts-file\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:45:59.646309 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646222    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8053a33-e1bf-43f4-b23b-34694e7ce042-iptables-alerter-script\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:45:59.646309 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646227    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.646309 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646260    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:59.646410 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646335    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-789l8\" (UniqueName: \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-kube-api-access-789l8\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.646410 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646398    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-tuned\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.646470 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646417    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba6439a7-7042-4edf-aa9c-6d33b168873d-tmp\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.646470 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646443    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a8053a33-e1bf-43f4-b23b-34694e7ce042-iptables-alerter-script\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:45:59.646535 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646473    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-cni-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.646535 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646482    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-tuned\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.646535 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646478    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:59.646535 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646495    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba6439a7-7042-4edf-aa9c-6d33b168873d-tmp\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.646713 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646577    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-k8s-cni-cncf-io\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.646713 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646619    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.646713 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646638    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6897566-da42-4b8d-ba85-7ec304624f2b-ovn-node-metrics-cert\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.646713 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646661    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-sys-fs\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.646713 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646684    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcxs7\" (UniqueName: \"kubernetes.io/projected/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-kube-api-access-lcxs7\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:59.646713 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646713    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:59.647054 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646735    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.647054 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646754    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.647054 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646799    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.647054 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646820    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.647054 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646825    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh9mp\" (UniqueName: \"kubernetes.io/projected/f6897566-da42-4b8d-ba85-7ec304624f2b-kube-api-access-dh9mp\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.647054 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646915    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f6897566-da42-4b8d-ba85-7ec304624f2b-ovn-node-metrics-cert\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.647054 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646947    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.647054 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646963    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dccd197b-ec88-445b-8de9-ff83832112bd-host\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:45:59.647054 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646979    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.647054 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.646997    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9whn4\" (UniqueName: \"kubernetes.io/projected/36c3843d-81b7-4554-80ab-f6fc5901f488-kube-api-access-9whn4\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.647054 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647009    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:59.647054 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647023    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.647054 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647048    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w7lh\" (UniqueName: \"kubernetes.io/projected/963f75f5-2539-456a-ae78-267d5b002bab-kube-api-access-8w7lh\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647071    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfvck\" (UniqueName: \"kubernetes.io/projected/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-kube-api-access-sfvck\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647096    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647123    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-cni-netd\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647148    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647173    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-etc-kubernetes\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647199    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89288ada-24c5-46af-ad86-7d21935ca939-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647211    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647253    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647313    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-systemd-units\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647358    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e-agent-certs\") pod \"konnectivity-agent-fd9xb\" (UID: \"d08a5487-7eaa-4438-8fcd-3ead8dc6c20e\") " pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647362    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647408    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/408c5c75-c039-4dc1-89c7-6b0f4412aa59-kubelet-config\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647433    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89288ada-24c5-46af-ad86-7d21935ca939-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647443    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647461    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647567    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysctl-d\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647587    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-system-cni-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647609    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-cni-multus\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647622    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e-agent-certs\") pod \"konnectivity-agent-fd9xb\" (UID: \"d08a5487-7eaa-4438-8fcd-3ead8dc6c20e\") " pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647632    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-systemd\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.647674 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647669    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647708    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-grpc-tls\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.647731    5893 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647744    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647776    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/408c5c75-c039-4dc1-89c7-6b0f4412aa59-original-pull-secret\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.647806    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:00.147777476 +0000 UTC m=+1.731207148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"service-ca" not registered
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647819    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sllfq\" (UniqueName: \"kubernetes.io/projected/b0eab2da-9812-424c-9be0-af423308d43c-kube-api-access-sllfq\") pod \"ingress-canary-dlv29\" (UID: \"b0eab2da-9812-424c-9be0-af423308d43c\") " pod="openshift-ingress-canary/ingress-canary-dlv29"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647842    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-client-ca-bundle\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647868    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/89288ada-24c5-46af-ad86-7d21935ca939-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647893    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647918    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-cni-binary-copy\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647944    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647971    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-config-out\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647972    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.647994    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-secret-metrics-server-client-certs\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648019    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/963f75f5-2539-456a-ae78-267d5b002bab-audit-log\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648035    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-grpc-tls\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648053    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-cni-binary-copy\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648082    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-client-ca-bundle\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648118    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/963f75f5-2539-456a-ae78-267d5b002bab-audit-log\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648130    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/89288ada-24c5-46af-ad86-7d21935ca939-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648144    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-config-out\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648173    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-cni-binary-copy\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648189    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-kubelet\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648216    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-cni-binary-copy\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648228    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648256    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-web-config\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648254    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-secret-metrics-server-client-certs\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648306    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-etc-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648327    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648374    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c8c3435-b6de-4aea-b596-970c9e17985c-tmp-dir\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648422    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648433    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-config-volume\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648458    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cljmm\" (UniqueName: \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-kube-api-access-cljmm\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648499    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c8c3435-b6de-4aea-b596-970c9e17985c-tmp-dir\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648504    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-web-config\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648544    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-systemd\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.648625 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648627    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-hostroot\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648667    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-config-volume\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648667    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-daemon-config\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648763    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-metrics-client-ca\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648799    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-daemon-config\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648839    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8c3435-b6de-4aea-b596-970c9e17985c-config-volume\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648867    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4zvz\" (UniqueName: \"kubernetes.io/projected/9b17184b-71db-4557-ae32-062ae53a9d53-kube-api-access-b4zvz\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648901    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35840c58-b7a6-438e-aed5-6c0d9764a793-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-krk2q\" (UID: \"35840c58-b7a6-438e-aed5-6c0d9764a793\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648905    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-metrics-client-ca\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648926    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-conf-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648952    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.648978    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649004    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-secret-metrics-server-tls\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649029    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649055    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649091    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/35840c58-b7a6-438e-aed5-6c0d9764a793-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-krk2q\" (UID: \"35840c58-b7a6-438e-aed5-6c0d9764a793\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649102    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6c8c3435-b6de-4aea-b596-970c9e17985c-config-volume\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.649117    5893 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649129    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649157    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-slash\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.649185    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:00.149168511 +0000 UTC m=+1.732598204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649227    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649230    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649249    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/963f75f5-2539-456a-ae78-267d5b002bab-secret-metrics-server-tls\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649256    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysconfig\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649246    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.649327    5893 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649360    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-lib-modules\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649389    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9b17184b-71db-4557-ae32-062ae53a9d53-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649401    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649444    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79d36201-724a-4cea-be2b-2e23dbf45682-tmp\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649486    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9b17184b-71db-4557-ae32-062ae53a9d53-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.649519    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:00.149508071 +0000 UTC m=+1.732937743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649504    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq4cv\" (UniqueName: \"kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv\") pod \"network-check-target-7qcht\" (UID: \"34fc63c2-b395-4702-a2e4-134161c0d50c\") " pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649575    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79d36201-724a-4cea-be2b-2e23dbf45682-tmp\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649584    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649619    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skwdd\" (UniqueName: \"kubernetes.io/projected/ba6439a7-7042-4edf-aa9c-6d33b168873d-kube-api-access-skwdd\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649656    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b17184b-71db-4557-ae32-062ae53a9d53-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649693    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649711    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-crio-socket\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649734    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkrz\" (UniqueName: \"kubernetes.io/projected/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-kube-api-access-6vkrz\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649751    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649766    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-node-log\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649854    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649883    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649898    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-sys\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649913    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649928    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649941    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-socket-dir-parent\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649963    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649980    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afb494b7-75a0-400e-be16-74d8d6882460-metrics-client-ca\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649992    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b17184b-71db-4557-ae32-062ae53a9d53-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.649998    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-cnibin\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650099    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8053a33-e1bf-43f4-b23b-34694e7ce042-host-slash\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650135    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87bm\" (UniqueName: \"kubernetes.io/projected/35840c58-b7a6-438e-aed5-6c0d9764a793-kube-api-access-b87bm\") pod \"monitoring-plugin-77f4f76594-krk2q\" (UID: \"35840c58-b7a6-438e-aed5-6c0d9764a793\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650225    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-metrics-certs\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:59.650203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650232    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-ovnkube-config\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650294    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650345    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650402    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-registration-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650449    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpcj9\" (UniqueName: \"kubernetes.io/projected/89288ada-24c5-46af-ad86-7d21935ca939-kube-api-access-mpcj9\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650503    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650551    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-tls\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650578    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650623    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkwhq\" (UniqueName: \"kubernetes.io/projected/6c8c3435-b6de-4aea-b596-970c9e17985c-kube-api-access-rkwhq\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650654    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-accelerators-collector-config\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650685    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x2pz\" (UniqueName: \"kubernetes.io/projected/afb494b7-75a0-400e-be16-74d8d6882460-kube-api-access-9x2pz\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650681    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/afb494b7-75a0-400e-be16-74d8d6882460-metrics-client-ca\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650728    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f6897566-da42-4b8d-ba85-7ec304624f2b-ovnkube-config\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650739    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-system-cni-dir\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.650734    5893 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650745    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650816    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650888    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-accelerators-collector-config\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650929    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.650957    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:00.150943101 +0000 UTC m=+1.734372796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650953    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-tls\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.650994    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651020    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c8c3435-b6de-4aea-b596-970c9e17985c-metrics-tls\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651054    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-sys\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651059    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651155    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-run\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651200    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm9qn\" (UniqueName: \"kubernetes.io/projected/a8053a33-e1bf-43f4-b23b-34694e7ce042-kube-api-access-sm9qn\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651214    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651239    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651264    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c8c3435-b6de-4aea-b596-970c9e17985c-metrics-tls\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651307    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651329    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651352    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651377    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-textfile\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651401    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651410    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651429    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86mph\" (UniqueName: \"kubernetes.io/projected/52f5c091-40a2-46e8-9214-ddf02da33d0b-kube-api-access-86mph\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651432    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b17184b-71db-4557-ae32-062ae53a9d53-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651436    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651453    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0eab2da-9812-424c-9be0-af423308d43c-cert\") pod \"ingress-canary-dlv29\" (UID: \"b0eab2da-9812-424c-9be0-af423308d43c\") " pod="openshift-ingress-canary/ingress-canary-dlv29"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651469    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-textfile\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651486    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-cnibin\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651488    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651512    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-kubelet\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.651543    5893 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651544    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-tls\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651550    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651579    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.651600    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:00.151585763 +0000 UTC m=+1.735015462 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651623    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0eab2da-9812-424c-9be0-af423308d43c-cert\") pod \"ingress-canary-dlv29\" (UID: \"b0eab2da-9812-424c-9be0-af423308d43c\") " pod="openshift-ingress-canary/ingress-canary-dlv29"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651629    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651652    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651668    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/963f75f5-2539-456a-ae78-267d5b002bab-metrics-server-audit-profiles\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651687    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-netns\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651756    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-tls\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651755    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/429e3a90-1868-4632-8631-5c0207978063-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651793    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-data-volume\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651814    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-tmp\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651825    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651833    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-log-socket\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651881    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-data-volume\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651914    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-tmp\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651928    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/963f75f5-2539-456a-ae78-267d5b002bab-metrics-server-audit-profiles\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651939    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-root\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651961    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-var-lib-kubelet\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.651988    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-multus-certs\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.652014    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.652033    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rml\" (UniqueName: \"kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.652048    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e-konnectivity-ca\") pod \"konnectivity-agent-fd9xb\" (UID: \"d08a5487-7eaa-4438-8fcd-3ead8dc6c20e\") " pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.652064    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/963f75f5-2539-456a-ae78-267d5b002bab-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.652081    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.652121    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.652157    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.652198    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/963f75f5-2539-456a-ae78-267d5b002bab-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.652246    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d08a5487-7eaa-4438-8fcd-3ead8dc6c20e-konnectivity-ca\") pod \"konnectivity-agent-fd9xb\" (UID: \"d08a5487-7eaa-4438-8fcd-3ead8dc6c20e\") " pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.652266    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.652475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.652384    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.667980 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.667953    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlm6t\" (UniqueName: \"kubernetes.io/projected/dccd197b-ec88-445b-8de9-ff83832112bd-kube-api-access-rlm6t\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:45:59.668080 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.667953    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8s98\" (UniqueName: \"kubernetes.io/projected/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-kube-api-access-x8s98\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.672319 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.672299    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm2ml\" (UniqueName: \"kubernetes.io/projected/429e3a90-1868-4632-8631-5c0207978063-kube-api-access-wm2ml\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.672454 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.672377    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcxs7\" (UniqueName: \"kubernetes.io/projected/825a2efb-6a8c-4bdd-bf89-cca7ee710f8e-kube-api-access-lcxs7\") pod \"network-metrics-daemon-zfp4t\" (UID: \"825a2efb-6a8c-4bdd-bf89-cca7ee710f8e\") " pod="openshift-multus/network-metrics-daemon-zfp4t"
Oct 30 12:45:59.672932 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.672907    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skwdd\" (UniqueName: \"kubernetes.io/projected/ba6439a7-7042-4edf-aa9c-6d33b168873d-kube-api-access-skwdd\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.673010 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.672992    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9whn4\" (UniqueName: \"kubernetes.io/projected/36c3843d-81b7-4554-80ab-f6fc5901f488-kube-api-access-9whn4\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.673073 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.673023    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sllfq\" (UniqueName: \"kubernetes.io/projected/b0eab2da-9812-424c-9be0-af423308d43c-kube-api-access-sllfq\") pod \"ingress-canary-dlv29\" (UID: \"b0eab2da-9812-424c-9be0-af423308d43c\") " pod="openshift-ingress-canary/ingress-canary-dlv29"
Oct 30 12:45:59.673122 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.673068    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-789l8\" (UniqueName: \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-kube-api-access-789l8\") pod \"prometheus-k8s-0\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:45:59.673157 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.673142    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cljmm\" (UniqueName: \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-kube-api-access-cljmm\") pod \"alertmanager-main-0\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:45:59.677547 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.677410    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86mph\" (UniqueName: \"kubernetes.io/projected/52f5c091-40a2-46e8-9214-ddf02da33d0b-kube-api-access-86mph\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:45:59.677686 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.677544    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh9mp\" (UniqueName: \"kubernetes.io/projected/f6897566-da42-4b8d-ba85-7ec304624f2b-kube-api-access-dh9mp\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.677873 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.677850    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq4cv\" (UniqueName: \"kubernetes.io/projected/34fc63c2-b395-4702-a2e4-134161c0d50c-kube-api-access-gq4cv\") pod \"network-check-target-7qcht\" (UID: \"34fc63c2-b395-4702-a2e4-134161c0d50c\") " pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:45:59.678032 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.678000    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4zvz\" (UniqueName: \"kubernetes.io/projected/9b17184b-71db-4557-ae32-062ae53a9d53-kube-api-access-b4zvz\") pod \"kube-state-metrics-5dc8d7d89b-fkq8t\" (UID: \"9b17184b-71db-4557-ae32-062ae53a9d53\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-fkq8t"
Oct 30 12:45:59.678119 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.678079    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkwhq\" (UniqueName: \"kubernetes.io/projected/6c8c3435-b6de-4aea-b596-970c9e17985c-kube-api-access-rkwhq\") pod \"dns-default-rm7hq\" (UID: \"6c8c3435-b6de-4aea-b596-970c9e17985c\") " pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:45:59.678119 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.678098    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkrz\" (UniqueName: \"kubernetes.io/projected/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-kube-api-access-6vkrz\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:59.678236 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.678197    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm9qn\" (UniqueName: \"kubernetes.io/projected/a8053a33-e1bf-43f4-b23b-34694e7ce042-kube-api-access-sm9qn\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:45:59.678236 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.678199    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpcj9\" (UniqueName: \"kubernetes.io/projected/89288ada-24c5-46af-ad86-7d21935ca939-kube-api-access-mpcj9\") pod \"openshift-state-metrics-7bd7489957-p6cmw\" (UID: \"89288ada-24c5-46af-ad86-7d21935ca939\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6cmw"
Oct 30 12:45:59.678415 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.678399    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x2pz\" (UniqueName: \"kubernetes.io/projected/afb494b7-75a0-400e-be16-74d8d6882460-kube-api-access-9x2pz\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.678448 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.678419    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfvck\" (UniqueName: \"kubernetes.io/projected/3d4edeee-f5a5-497d-a53c-2a6c82a3fe85-kube-api-access-sfvck\") pod \"thanos-querier-5ff77c7974-6cgw8\" (UID: \"3d4edeee-f5a5-497d-a53c-2a6c82a3fe85\") " pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:45:59.678481 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.678461    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w7lh\" (UniqueName: \"kubernetes.io/projected/963f75f5-2539-456a-ae78-267d5b002bab-kube-api-access-8w7lh\") pod \"metrics-server-b6fc5f989-jm9qh\" (UID: \"963f75f5-2539-456a-ae78-267d5b002bab\") " pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:45:59.680864 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.680847    5893 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 12:45:59.680915 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.680867    5893 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 12:45:59.680915 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.680876    5893 projected.go:196] Error preparing data for projected volume kube-api-access-c6rml for pod openshift-console/console-7d587d759d-h9f5b: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 12:45:59.680973 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:45:59.680934    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:00.180922833 +0000 UTC m=+1.764352520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-c6rml" (UniqueName: "kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 12:45:59.681046 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.681031    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87bm\" (UniqueName: \"kubernetes.io/projected/35840c58-b7a6-438e-aed5-6c0d9764a793-kube-api-access-b87bm\") pod \"monitoring-plugin-77f4f76594-krk2q\" (UID: \"35840c58-b7a6-438e-aed5-6c0d9764a793\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:45:59.752806 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752725    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dccd197b-ec88-445b-8de9-ff83832112bd-host\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:45:59.752806 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752758    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-cni-netd\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.752806 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752777    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752822    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-cni-netd\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752826    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/dccd197b-ec88-445b-8de9-ff83832112bd-host\") pod \"node-ca-bzkw4\" (UID: \"dccd197b-ec88-445b-8de9-ff83832112bd\") " pod="openshift-image-registry/node-ca-bzkw4"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752844    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-etc-kubernetes\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752861    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-systemd-units\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752868    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752877    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/408c5c75-c039-4dc1-89c7-6b0f4412aa59-kubelet-config\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752896    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysctl-d\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752906    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-etc-kubernetes\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752915    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-systemd-units\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752927    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/408c5c75-c039-4dc1-89c7-6b0f4412aa59-kubelet-config\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752913    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-system-cni-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752951    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-system-cni-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752964    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-cni-multus\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.752986    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-systemd\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753020    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-cni-multus\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753051    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-systemd\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753022    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysctl-d\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753026    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753082    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753110    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-kubelet\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753142    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-etc-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753173    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-systemd\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753197    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-hostroot\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753199    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-kubelet\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753206    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-etc-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753242    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-systemd\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753254    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-hostroot\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753259    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-conf-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753319    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-slash\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753328    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-conf-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753342    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysconfig\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753363    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-lib-modules\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753363    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-slash\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753395    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-crio-socket\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753408    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysconfig\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753417    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-node-log\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753443    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753458    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fb5dfe9c-5cf0-43af-875c-3bc0fd07c265-crio-socket\") pod \"insights-runtime-extractor-k47jk\" (UID: \"fb5dfe9c-5cf0-43af-875c-3bc0fd07c265\") " pod="openshift-insights/insights-runtime-extractor-k47jk"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753465    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-sys\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753481    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-lib-modules\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753488    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753503    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753512    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-socket-dir-parent\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753518    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-sys\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753536    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-node-log\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753554    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-socket-dir-parent\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753559    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753582    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-cnibin\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753605    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8053a33-e1bf-43f4-b23b-34694e7ce042-host-slash\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753623    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753634    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a8053a33-e1bf-43f4-b23b-34694e7ce042-host-slash\") pod \"iptables-alerter-jtfpn\" (UID: \"a8053a33-e1bf-43f4-b23b-34694e7ce042\") " pod="openshift-network-operator/iptables-alerter-jtfpn"
Oct 30 12:45:59.753624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753637    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-registration-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753679    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-cnibin\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753689    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-system-cni-dir\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753699    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-registration-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753699    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-kubelet-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753730    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-sys\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753728    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/429e3a90-1868-4632-8631-5c0207978063-system-cni-dir\") pod \"multus-additional-cni-plugins-jzgc2\" (UID: \"429e3a90-1868-4632-8631-5c0207978063\") " pod="openshift-multus/multus-additional-cni-plugins-jzgc2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753759    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-sys\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753788    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-run\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753814    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-run\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753841    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-cnibin\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753860    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-kubelet\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753875    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753891    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-netns\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753906    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-log-socket\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753920    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-root\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753933    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-kubelet\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753953    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-cnibin\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753934    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-var-lib-kubelet\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753963    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-netns\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753977    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-log-socket\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753982    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-etc-selinux\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753971    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-root\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753988    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-multus-certs\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.753996    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-var-lib-kubelet\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754016    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-multus-certs\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754047    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-var-lib-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754067    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-host\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754084    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-var-lib-openvswitch\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754087    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-cni-bin\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754129    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-host\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754141    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-cni-bin\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754164    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-modprobe-d\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754114    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-var-lib-cni-bin\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754189    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-socket-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754211    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-device-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754215    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-cni-bin\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754226    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/408c5c75-c039-4dc1-89c7-6b0f4412aa59-dbus\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754240    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysctl-conf\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754250    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-socket-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754256    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-run-netns\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754264    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-modprobe-d\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754300    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-wtmp\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754305    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/408c5c75-c039-4dc1-89c7-6b0f4412aa59-dbus\") pod \"global-pull-secret-syncer-79f2f\" (UID: \"408c5c75-c039-4dc1-89c7-6b0f4412aa59\") " pod="kube-system/global-pull-secret-syncer-79f2f"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754306    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-device-dir\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754320    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-kubernetes\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754328    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-host-run-netns\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754342    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-ovn\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754364    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52f5c091-40a2-46e8-9214-ddf02da33d0b-hosts-file\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754376    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-kubernetes\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754381    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/afb494b7-75a0-400e-be16-74d8d6882460-node-exporter-wtmp\") pod \"node-exporter-2xmlk\" (UID: \"afb494b7-75a0-400e-be16-74d8d6882460\") " pod="openshift-monitoring/node-exporter-2xmlk"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754391    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ba6439a7-7042-4edf-aa9c-6d33b168873d-etc-sysctl-conf\") pod \"tuned-gxdv2\" (UID: \"ba6439a7-7042-4edf-aa9c-6d33b168873d\") " pod="openshift-cluster-node-tuning-operator/tuned-gxdv2"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754392    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-cni-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754388    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f6897566-da42-4b8d-ba85-7ec304624f2b-run-ovn\") pod \"ovnkube-node-dg4xp\" (UID: \"f6897566-da42-4b8d-ba85-7ec304624f2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754429    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/52f5c091-40a2-46e8-9214-ddf02da33d0b-hosts-file\") pod \"node-resolver-tg5ns\" (UID: \"52f5c091-40a2-46e8-9214-ddf02da33d0b\") " pod="openshift-dns/node-resolver-tg5ns"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754433    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-k8s-cni-cncf-io\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754447    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-multus-cni-dir\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754462    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-sys-fs\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754478    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bbdf052-fa04-4e1c-983a-04b313a0cdfb-host-run-k8s-cni-cncf-io\") pod \"multus-x48w4\" (UID: \"9bbdf052-fa04-4e1c-983a-04b313a0cdfb\") " pod="openshift-multus/multus-x48w4"
Oct 30 12:45:59.754859 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:45:59.754528    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/36c3843d-81b7-4554-80ab-f6fc5901f488-sys-fs\") pod \"aws-ebs-csi-driver-node-dmrrv\" (UID: \"36c3843d-81b7-4554-80ab-f6fc5901f488\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-dmrrv"
Oct 30 12:46:00.158547 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:00.158458    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:00.158547 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:00.158501    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:00.158547 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:00.158529    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:00.158815 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:00.158561    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:00.158815 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.158584    5893 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 12:46:00.158815 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.158647    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:01.158633721 +0000 UTC m=+2.742063405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 12:46:00.158815 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.158647    5893 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 12:46:00.158815 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.158653    5893 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 12:46:00.158815 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:00.158666    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:00.158815 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.158674    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:01.158667816 +0000 UTC m=+2.742097551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 12:46:00.158815 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.158584    5893 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 12:46:00.158815 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.158718    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:01.158704244 +0000 UTC m=+2.742133938 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 12:46:00.158815 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.158761    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:01.158746793 +0000 UTC m=+2.742176487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 12:46:00.158815 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.158786    5893 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 12:46:00.158815 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.158815    5893 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 12:46:00.159205 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.158838    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:01.158831022 +0000 UTC m=+2.742260706 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"service-ca" not registered
Oct 30 12:46:00.159205 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.158851    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:01.158846152 +0000 UTC m=+2.742275823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-config" not registered
Oct 30 12:46:00.159205 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:00.158787    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:00.259679 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:00.259647    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rml\" (UniqueName: \"kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:00.259848 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.259816    5893 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 12:46:00.259848 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.259838    5893 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 12:46:00.259848 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.259847    5893 projected.go:196] Error preparing data for projected volume kube-api-access-c6rml for pod openshift-console/console-7d587d759d-h9f5b: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 12:46:00.259958 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:00.259898    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:01.25988466 +0000 UTC m=+2.843314347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-c6rml" (UniqueName: "kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 12:46:01.169364 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:01.169327    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:01.169375    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:01.169414    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:01.169464    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:01.169492    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.169498    5893 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:01.169520    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.169523    5893 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.169552    5893 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.169562    5893 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.169569    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:03.169554663 +0000 UTC m=+4.752984369 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.169623    5893 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.169634    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:03.169622349 +0000 UTC m=+4.753052019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.169566    5893 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.169708    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:03.169691691 +0000 UTC m=+4.753121368 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-config" not registered
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.169726    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:03.169717075 +0000 UTC m=+4.753146746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"service-ca" not registered
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.169738    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:03.169731583 +0000 UTC m=+4.753161259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 12:46:01.169786 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.169750    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:03.169743917 +0000 UTC m=+4.753173588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 12:46:01.270647 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:01.270615    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rml\" (UniqueName: \"kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:01.270814 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.270767    5893 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 12:46:01.270814 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.270783    5893 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 12:46:01.270814 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.270792    5893 projected.go:196] Error preparing data for projected volume kube-api-access-c6rml for pod openshift-console/console-7d587d759d-h9f5b: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 12:46:01.270925 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:01.270838    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:03.27082573 +0000 UTC m=+4.854255414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-c6rml" (UniqueName: "kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 12:46:01.477799 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:01.477770    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rm7hq"
Oct 30 12:46:01.623249 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:01.623226    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:01.730634 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:01.730568    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5ff77c7974-6cgw8"
Oct 30 12:46:02.603889 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:02.603859    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:46:03.191971 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:03.191934    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:03.192248 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:03.191991    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:03.192248 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.192078    5893 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 12:46:03.192248 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.192081    5893 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 12:46:03.192248 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:03.192132    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:03.192248 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.192154    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:07.192133879 +0000 UTC m=+8.775563564 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-config" not registered
Oct 30 12:46:03.192248 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.192185    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:07.19217611 +0000 UTC m=+8.775605784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"service-ca" not registered
Oct 30 12:46:03.192248 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.192188    5893 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 12:46:03.192248 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:03.192229    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:03.192248 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.192238    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:07.192224589 +0000 UTC m=+8.775654277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 12:46:03.192681 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.192295    5893 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 12:46:03.192681 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:03.192305    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:03.192681 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.192329    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:07.192318889 +0000 UTC m=+8.775748575 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 12:46:03.192681 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.192371    5893 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 12:46:03.192681 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:03.192370    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:03.192681 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.192423    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:07.19241472 +0000 UTC m=+8.775844412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 12:46:03.192681 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.192435    5893 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 12:46:03.192681 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.192470    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:07.192459017 +0000 UTC m=+8.775888704 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 12:46:03.293106 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:03.293071    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rml\" (UniqueName: \"kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:03.293262 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.293198    5893 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 12:46:03.293262 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.293217    5893 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 12:46:03.293262 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.293227    5893 projected.go:196] Error preparing data for projected volume kube-api-access-c6rml for pod openshift-console/console-7d587d759d-h9f5b: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 12:46:03.293396 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:03.293293    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:07.293261298 +0000 UTC m=+8.876690986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-c6rml" (UniqueName: "kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 12:46:04.766672 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:04.766643    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:46:07.237537 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:07.237501    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:07.237561    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:07.237596    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:07.237622    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:07.237640    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.237659    5893 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:07.237673    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.237712    5893 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.237716    5893 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.237734    5893 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.237757    5893 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.237774    5893 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.237738    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:15.237717091 +0000 UTC m=+16.821146779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.237809    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:15.237792087 +0000 UTC m=+16.821221759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"service-ca" not registered
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.237827    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:15.23781502 +0000 UTC m=+16.821244715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.237847    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:15.237841378 +0000 UTC m=+16.821271049 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.237859    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:15.237854019 +0000 UTC m=+16.821283690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-config" not registered
Oct 30 12:46:07.238055 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.237874    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:15.237868658 +0000 UTC m=+16.821298329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 12:46:07.338946 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:07.338902    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rml\" (UniqueName: \"kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:07.339125 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.339082    5893 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 12:46:07.339125 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.339105    5893 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 12:46:07.339125 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.339115    5893 projected.go:196] Error preparing data for projected volume kube-api-access-c6rml for pod openshift-console/console-7d587d759d-h9f5b: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 12:46:07.339225 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:07.339165    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:15.339151857 +0000 UTC m=+16.922581545 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-c6rml" (UniqueName: "kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 12:46:07.465566 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:07.465529    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:46:07.465770 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:07.465758    5893 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 12:46:07.868536 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:07.868510    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-krk2q"
Oct 30 12:46:08.958204 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:08.958136    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dg4xp"
Oct 30 12:46:15.319597 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:15.319553    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:15.319597 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:15.319614    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:15.319635    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:15.319663    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:15.319689    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.319693    5893 configmap.go:193] Couldn't get configMap openshift-console/service-ca: object "openshift-console"/"service-ca" not registered
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.319723    5893 configmap.go:193] Couldn't get configMap openshift-console/trusted-ca-bundle: object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.319765    5893 configmap.go:193] Couldn't get configMap openshift-console/console-config: object "openshift-console"/"console-config" not registered
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.319695    5893 configmap.go:193] Couldn't get configMap openshift-console/oauth-serving-cert: object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:15.319727    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.319784    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:31.319765016 +0000 UTC m=+32.903194706 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"service-ca" not registered
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.319816    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:31.319806755 +0000 UTC m=+32.903236433 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"trusted-ca-bundle" not registered
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.319824    5893 secret.go:189] Couldn't get secret openshift-console/console-serving-cert: object "openshift-console"/"console-serving-cert" not registered
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.319837    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:31.3198285 +0000 UTC m=+32.903258172 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-config" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-config" not registered
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.319837    5893 secret.go:189] Couldn't get secret openshift-console/console-oauth-config: object "openshift-console"/"console-oauth-config" not registered
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.319854    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:31.319846328 +0000 UTC m=+32.903276003 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "oauth-serving-cert" (UniqueName: "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"oauth-serving-cert" not registered
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.319879    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:31.319871722 +0000 UTC m=+32.903301396 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-serving-cert" (UniqueName: "kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-serving-cert" not registered
Oct 30 12:46:15.320076 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.319945    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:31.31992675 +0000 UTC m=+32.903356434 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "console-oauth-config" (UniqueName: "kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : object "openshift-console"/"console-oauth-config" not registered
Oct 30 12:46:15.420717 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:15.420680    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rml\" (UniqueName: \"kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml\") pod \"console-7d587d759d-h9f5b\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") " pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:15.420877 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.420840    5893 projected.go:291] Couldn't get configMap openshift-console/kube-root-ca.crt: object "openshift-console"/"kube-root-ca.crt" not registered
Oct 30 12:46:15.420877 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.420859    5893 projected.go:291] Couldn't get configMap openshift-console/openshift-service-ca.crt: object "openshift-console"/"openshift-service-ca.crt" not registered
Oct 30 12:46:15.420877 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.420870    5893 projected.go:196] Error preparing data for projected volume kube-api-access-c6rml for pod openshift-console/console-7d587d759d-h9f5b: [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 12:46:15.420969 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:15.420922    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml podName:79d36201-724a-4cea-be2b-2e23dbf45682 nodeName:}" failed. No retries permitted until 2026-10-30 12:46:31.420907876 +0000 UTC m=+33.004337562 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-c6rml" (UniqueName: "kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml") pod "console-7d587d759d-h9f5b" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682") : [object "openshift-console"/"kube-root-ca.crt" not registered, object "openshift-console"/"openshift-service-ca.crt" not registered]
Oct 30 12:46:15.711402 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:15.711330    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-b6fc5f989-jm9qh"
Oct 30 12:46:18.642928 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:18.642891    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:46:21.943615 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:21.943579    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:46:21.944051 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:21.943854    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="alertmanager" containerID="cri-o://087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494" gracePeriod=120
Oct 30 12:46:21.944051 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:21.943929    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="kube-rbac-proxy-metric" containerID="cri-o://7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378" gracePeriod=120
Oct 30 12:46:21.944051 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:21.943945    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="prom-label-proxy" containerID="cri-o://4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb" gracePeriod=120
Oct 30 12:46:21.944221 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:21.944058    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="kube-rbac-proxy" containerID="cri-o://6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb" gracePeriod=120
Oct 30 12:46:21.944221 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:21.944058    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="config-reloader" containerID="cri-o://4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747" gracePeriod=120
Oct 30 12:46:21.944221 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:21.944078    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="kube-rbac-proxy-web" containerID="cri-o://e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218" gracePeriod=120
Oct 30 12:46:22.686585 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:22.686553    5893 generic.go:356] "Generic (PLEG): container finished" podID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerID="4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb" exitCode=0
Oct 30 12:46:22.686585 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:22.686578    5893 generic.go:356] "Generic (PLEG): container finished" podID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerID="6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb" exitCode=0
Oct 30 12:46:22.686585 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:22.686585    5893 generic.go:356] "Generic (PLEG): container finished" podID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerID="4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747" exitCode=0
Oct 30 12:46:22.686585 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:22.686593    5893 generic.go:356] "Generic (PLEG): container finished" podID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerID="087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494" exitCode=0
Oct 30 12:46:22.686860 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:22.686602    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerDied","Data":"4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb"}
Oct 30 12:46:22.686860 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:22.686663    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerDied","Data":"6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb"}
Oct 30 12:46:22.686860 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:22.686691    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerDied","Data":"4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747"}
Oct 30 12:46:22.686860 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:22.686716    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerDied","Data":"087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494"}
Oct 30 12:46:23.204035 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.204010    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.299394 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.299316    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-tls-assets\" (UniqueName: \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-tls-assets\") pod \"63c94b3e-9026-4fba-9788-03f2d981b53b\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") "
Oct 30 12:46:23.299394 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.299358    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-kube-api-access-cljmm\" (UniqueName: \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-kube-api-access-cljmm\") pod \"63c94b3e-9026-4fba-9788-03f2d981b53b\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") "
Oct 30 12:46:23.299614 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.299405    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-cluster-tls-config\") pod \"63c94b3e-9026-4fba-9788-03f2d981b53b\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") "
Oct 30 12:46:23.299614 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.299463    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-metrics-client-ca\") pod \"63c94b3e-9026-4fba-9788-03f2d981b53b\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") "
Oct 30 12:46:23.299614 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.299508    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-trusted-ca-bundle\") pod \"63c94b3e-9026-4fba-9788-03f2d981b53b\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") "
Oct 30 12:46:23.299614 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.299536    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-web\") pod \"63c94b3e-9026-4fba-9788-03f2d981b53b\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") "
Oct 30 12:46:23.299614 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.299575    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-config-out\") pod \"63c94b3e-9026-4fba-9788-03f2d981b53b\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") "
Oct 30 12:46:23.299614 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.299605    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"63c94b3e-9026-4fba-9788-03f2d981b53b\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") "
Oct 30 12:46:23.299929 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.299640    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-web-config\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-web-config\") pod \"63c94b3e-9026-4fba-9788-03f2d981b53b\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") "
Oct 30 12:46:23.299929 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.299680    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy\") pod \"63c94b3e-9026-4fba-9788-03f2d981b53b\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") "
Oct 30 12:46:23.299929 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.299720    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-config-volume\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-config-volume\") pod \"63c94b3e-9026-4fba-9788-03f2d981b53b\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") "
Oct 30 12:46:23.299929 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.299746    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-main-tls\") pod \"63c94b3e-9026-4fba-9788-03f2d981b53b\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") "
Oct 30 12:46:23.299929 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.299778    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-main-db\") pod \"63c94b3e-9026-4fba-9788-03f2d981b53b\" (UID: \"63c94b3e-9026-4fba-9788-03f2d981b53b\") "
Oct 30 12:46:23.300150 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.299949    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-trusted-ca-bundle" pod "63c94b3e-9026-4fba-9788-03f2d981b53b" (UID: "63c94b3e-9026-4fba-9788-03f2d981b53b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:23.300378 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.300350    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-metrics-client-ca" pod "63c94b3e-9026-4fba-9788-03f2d981b53b" (UID: "63c94b3e-9026-4fba-9788-03f2d981b53b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:23.300532 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.300417    5893 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.300846 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.300819    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-main-db" pod "63c94b3e-9026-4fba-9788-03f2d981b53b" (UID: "63c94b3e-9026-4fba-9788-03f2d981b53b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:23.303635 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.303576    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-kube-api-access-cljmm" pod "63c94b3e-9026-4fba-9788-03f2d981b53b" (UID: "63c94b3e-9026-4fba-9788-03f2d981b53b"). InnerVolumeSpecName "kube-api-access-cljmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:23.305127 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.305081    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-config-volume" pod "63c94b3e-9026-4fba-9788-03f2d981b53b" (UID: "63c94b3e-9026-4fba-9788-03f2d981b53b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:23.305477 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.305419    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-main-tls" pod "63c94b3e-9026-4fba-9788-03f2d981b53b" (UID: "63c94b3e-9026-4fba-9788-03f2d981b53b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:23.306452 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.306404    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-metric" pod "63c94b3e-9026-4fba-9788-03f2d981b53b" (UID: "63c94b3e-9026-4fba-9788-03f2d981b53b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:23.306554 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.306481    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy" pod "63c94b3e-9026-4fba-9788-03f2d981b53b" (UID: "63c94b3e-9026-4fba-9788-03f2d981b53b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:23.306554 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.306522    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-config-out" pod "63c94b3e-9026-4fba-9788-03f2d981b53b" (UID: "63c94b3e-9026-4fba-9788-03f2d981b53b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:23.306671 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.306623    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-web" pod "63c94b3e-9026-4fba-9788-03f2d981b53b" (UID: "63c94b3e-9026-4fba-9788-03f2d981b53b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:23.306986 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.306961    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-tls-assets" pod "63c94b3e-9026-4fba-9788-03f2d981b53b" (UID: "63c94b3e-9026-4fba-9788-03f2d981b53b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:23.311142 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.311109    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-cluster-tls-config" pod "63c94b3e-9026-4fba-9788-03f2d981b53b" (UID: "63c94b3e-9026-4fba-9788-03f2d981b53b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:23.317354 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.317322    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-web-config" pod "63c94b3e-9026-4fba-9788-03f2d981b53b" (UID: "63c94b3e-9026-4fba-9788-03f2d981b53b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:23.401879 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.401846    5893 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.401879 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.401877    5893 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-config-out\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.401879 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.401887    5893 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.402104 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.401897    5893 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-web-config\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.402104 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.401906    5893 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.402104 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.401914    5893 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.402104 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.401925    5893 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-secret-alertmanager-main-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.402104 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.401933    5893 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/63c94b3e-9026-4fba-9788-03f2d981b53b-alertmanager-main-db\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.402104 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.401942    5893 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-tls-assets\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.402104 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.401950    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cljmm\" (UniqueName: \"kubernetes.io/projected/63c94b3e-9026-4fba-9788-03f2d981b53b-kube-api-access-cljmm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.402104 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.401958    5893 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/63c94b3e-9026-4fba-9788-03f2d981b53b-cluster-tls-config\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.402104 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.401966    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63c94b3e-9026-4fba-9788-03f2d981b53b-metrics-client-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:23.693498 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.693420    5893 generic.go:356] "Generic (PLEG): container finished" podID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerID="7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378" exitCode=0
Oct 30 12:46:23.693498 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.693444    5893 generic.go:356] "Generic (PLEG): container finished" podID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerID="e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218" exitCode=0
Oct 30 12:46:23.693676 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.693502    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.693676 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.693502    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerDied","Data":"7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378"}
Oct 30 12:46:23.693676 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.693573    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerDied","Data":"e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218"}
Oct 30 12:46:23.693676 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.693600    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"63c94b3e-9026-4fba-9788-03f2d981b53b","Type":"ContainerDied","Data":"93bd4adc1974ea9c1a5ddfb97845c697f9be6f6b2f38715a630afc2f75214529"}
Oct 30 12:46:23.693676 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.693629    5893 scope.go:122] "RemoveContainer" containerID="4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb"
Oct 30 12:46:23.702295 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.702259    5893 scope.go:122] "RemoveContainer" containerID="7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378"
Oct 30 12:46:23.709407 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.709390    5893 scope.go:122] "RemoveContainer" containerID="6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb"
Oct 30 12:46:23.716253 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.716239    5893 scope.go:122] "RemoveContainer" containerID="e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218"
Oct 30 12:46:23.723135 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.723119    5893 scope.go:122] "RemoveContainer" containerID="4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747"
Oct 30 12:46:23.729832 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.729817    5893 scope.go:122] "RemoveContainer" containerID="087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494"
Oct 30 12:46:23.736940 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.736918    5893 scope.go:122] "RemoveContainer" containerID="09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6"
Oct 30 12:46:23.742661 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.742636    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:46:23.744788 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.744772    5893 scope.go:122] "RemoveContainer" containerID="4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb"
Oct 30 12:46:23.745074 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:23.745053    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb\": container with ID starting with 4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb not found: ID does not exist" containerID="4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb"
Oct 30 12:46:23.745151 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.745088    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb"} err="failed to get container status \"4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb\": rpc error: code = NotFound desc = could not find container \"4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb\": container with ID starting with 4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb not found: ID does not exist"
Oct 30 12:46:23.745151 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.745132    5893 scope.go:122] "RemoveContainer" containerID="7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378"
Oct 30 12:46:23.745396 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:23.745377    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378\": container with ID starting with 7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378 not found: ID does not exist" containerID="7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378"
Oct 30 12:46:23.745439 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.745404    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378"} err="failed to get container status \"7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378\": rpc error: code = NotFound desc = could not find container \"7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378\": container with ID starting with 7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378 not found: ID does not exist"
Oct 30 12:46:23.745439 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.745421    5893 scope.go:122] "RemoveContainer" containerID="6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb"
Oct 30 12:46:23.745625 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:23.745598    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb\": container with ID starting with 6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb not found: ID does not exist" containerID="6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb"
Oct 30 12:46:23.745678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.745625    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb"} err="failed to get container status \"6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb\": rpc error: code = NotFound desc = could not find container \"6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb\": container with ID starting with 6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb not found: ID does not exist"
Oct 30 12:46:23.745678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.745638    5893 scope.go:122] "RemoveContainer" containerID="e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218"
Oct 30 12:46:23.745862 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:23.745847    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218\": container with ID starting with e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218 not found: ID does not exist" containerID="e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218"
Oct 30 12:46:23.745910 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.745866    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218"} err="failed to get container status \"e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218\": rpc error: code = NotFound desc = could not find container \"e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218\": container with ID starting with e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218 not found: ID does not exist"
Oct 30 12:46:23.745910 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.745878    5893 scope.go:122] "RemoveContainer" containerID="4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747"
Oct 30 12:46:23.746174 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:23.746098    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747\": container with ID starting with 4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747 not found: ID does not exist" containerID="4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747"
Oct 30 12:46:23.746236 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.746180    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747"} err="failed to get container status \"4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747\": rpc error: code = NotFound desc = could not find container \"4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747\": container with ID starting with 4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747 not found: ID does not exist"
Oct 30 12:46:23.746236 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.746202    5893 scope.go:122] "RemoveContainer" containerID="087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494"
Oct 30 12:46:23.746469 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:23.746455    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494\": container with ID starting with 087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494 not found: ID does not exist" containerID="087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494"
Oct 30 12:46:23.746508 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.746474    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494"} err="failed to get container status \"087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494\": rpc error: code = NotFound desc = could not find container \"087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494\": container with ID starting with 087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494 not found: ID does not exist"
Oct 30 12:46:23.746508 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.746486    5893 scope.go:122] "RemoveContainer" containerID="09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6"
Oct 30 12:46:23.746683 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:23.746668    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6\": container with ID starting with 09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6 not found: ID does not exist" containerID="09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6"
Oct 30 12:46:23.746720 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.746687    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6"} err="failed to get container status \"09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6\": rpc error: code = NotFound desc = could not find container \"09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6\": container with ID starting with 09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6 not found: ID does not exist"
Oct 30 12:46:23.746720 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.746699    5893 scope.go:122] "RemoveContainer" containerID="4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb"
Oct 30 12:46:23.746943 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.746924    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb"} err="failed to get container status \"4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb\": rpc error: code = NotFound desc = could not find container \"4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb\": container with ID starting with 4e32e941e8a0366fd1e1c44364b25b9c189e28c9cf24b3d6d859957d2be790fb not found: ID does not exist"
Oct 30 12:46:23.746991 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.746944    5893 scope.go:122] "RemoveContainer" containerID="7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378"
Oct 30 12:46:23.747152 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.747137    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378"} err="failed to get container status \"7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378\": rpc error: code = NotFound desc = could not find container \"7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378\": container with ID starting with 7929591c11caf64e71739e3926c3ca894fd2cbf2cf88a5749fb609d9403d8378 not found: ID does not exist"
Oct 30 12:46:23.747200 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.747152    5893 scope.go:122] "RemoveContainer" containerID="6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb"
Oct 30 12:46:23.747354 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.747337    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb"} err="failed to get container status \"6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb\": rpc error: code = NotFound desc = could not find container \"6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb\": container with ID starting with 6fb11ae89e548a5933d1d08a5f0fc87f6e72ec49c36e58a170735b28c01af5bb not found: ID does not exist"
Oct 30 12:46:23.747407 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.747354    5893 scope.go:122] "RemoveContainer" containerID="e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218"
Oct 30 12:46:23.747564 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.747547    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218"} err="failed to get container status \"e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218\": rpc error: code = NotFound desc = could not find container \"e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218\": container with ID starting with e409f54b50e91022f2be7649222c378562c4fd3297638f5f6da42805b0ea0218 not found: ID does not exist"
Oct 30 12:46:23.747601 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.747564    5893 scope.go:122] "RemoveContainer" containerID="4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747"
Oct 30 12:46:23.747761 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.747746    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747"} err="failed to get container status \"4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747\": rpc error: code = NotFound desc = could not find container \"4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747\": container with ID starting with 4df1603128dacf258facf07c954649e7a812f0f0b3c2e0cac5206305a1ec1747 not found: ID does not exist"
Oct 30 12:46:23.747813 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.747761    5893 scope.go:122] "RemoveContainer" containerID="087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494"
Oct 30 12:46:23.747979 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.747964    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494"} err="failed to get container status \"087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494\": rpc error: code = NotFound desc = could not find container \"087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494\": container with ID starting with 087c23cb4073466d4afe5e2f4059049ae6af80c81684846de3861b6cbd789494 not found: ID does not exist"
Oct 30 12:46:23.747979 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.747978    5893 scope.go:122] "RemoveContainer" containerID="09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6"
Oct 30 12:46:23.748162 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.748148    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6"} err="failed to get container status \"09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6\": rpc error: code = NotFound desc = could not find container \"09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6\": container with ID starting with 09d09eb7ee2044c45509b55419d72d84ea7e2de5769d094360e0b90d829b4cd6 not found: ID does not exist"
Oct 30 12:46:23.752670 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.752650    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:46:23.778681 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.778656    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:46:23.779009 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.778993    5893 cpu_manager.go:414] "containerMap: removing container" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="kube-rbac-proxy-metric"
Oct 30 12:46:23.779115 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779012    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="kube-rbac-proxy-metric"
Oct 30 12:46:23.779115 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779032    5893 cpu_manager.go:414] "containerMap: removing container" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="kube-rbac-proxy"
Oct 30 12:46:23.779115 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779039    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="kube-rbac-proxy"
Oct 30 12:46:23.779115 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779049    5893 cpu_manager.go:414] "containerMap: removing container" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="alertmanager"
Oct 30 12:46:23.779115 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779057    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="alertmanager"
Oct 30 12:46:23.779115 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779065    5893 cpu_manager.go:414] "containerMap: removing container" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="init-config-reloader"
Oct 30 12:46:23.779115 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779072    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="init-config-reloader"
Oct 30 12:46:23.779115 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779109    5893 cpu_manager.go:414] "containerMap: removing container" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="config-reloader"
Oct 30 12:46:23.779115 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779116    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="config-reloader"
Oct 30 12:46:23.779564 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779127    5893 cpu_manager.go:414] "containerMap: removing container" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="kube-rbac-proxy-web"
Oct 30 12:46:23.779564 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779135    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="kube-rbac-proxy-web"
Oct 30 12:46:23.779564 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779155    5893 cpu_manager.go:414] "containerMap: removing container" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="prom-label-proxy"
Oct 30 12:46:23.779564 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779162    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="prom-label-proxy"
Oct 30 12:46:23.779564 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779238    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="config-reloader"
Oct 30 12:46:23.779564 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779252    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="kube-rbac-proxy-web"
Oct 30 12:46:23.779564 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779262    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="prom-label-proxy"
Oct 30 12:46:23.779564 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779286    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="kube-rbac-proxy-metric"
Oct 30 12:46:23.779564 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779299    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="kube-rbac-proxy"
Oct 30 12:46:23.779564 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779306    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="alertmanager"
Oct 30 12:46:23.779564 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.779315    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" containerName="init-config-reloader"
Oct 30 12:46:23.784887 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.784864    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.799648 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.799629    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 12:46:23.799763 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.799681    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 12:46:23.799763 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.799743    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 12:46:23.799894 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.799780    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 12:46:23.799894 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.799890    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 12:46:23.800012 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.799962    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 12:46:23.800012 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.799980    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 12:46:23.800170 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.800153    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5q4kn\""
Oct 30 12:46:23.800231 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.800160    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 12:46:23.810031 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.810008    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 12:46:23.811633 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.811614    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:46:23.906930 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.906896    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-config-volume\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.906930 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.906930    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.907144 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.906949    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.907144 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.906968    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.907211 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.907141    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.907211 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.907174    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.907211 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.907194    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-config-out\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.907333 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.907315    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-web-config\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.907380 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.907367    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.907415 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.907390    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgzc8\" (UniqueName: \"kubernetes.io/projected/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-kube-api-access-lgzc8\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.907415 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.907408    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.907514 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.907500    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:23.907546 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:23.907525    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.008580 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.008541    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.008580 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.008582    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.008779 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.008653    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.008779 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.008685    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.008779 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.008710    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-config-out\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.008779 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.008753    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-web-config\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.008967 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.008809    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.008967 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.008834    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgzc8\" (UniqueName: \"kubernetes.io/projected/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-kube-api-access-lgzc8\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.008967 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.008863    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.008967 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.008918    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.008967 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.008947    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.009179 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.009002    5893 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 12:46:24.009385 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.009361    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-config-volume\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.009492 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.009399    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.009743 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.009716    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.009909 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.009883    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.010118 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.010046    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.012173 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.012101    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.012857 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.012449    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.012857 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.012453    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-web-config\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.012857 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.012576    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-config-volume\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.012857 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.012574    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-config-out\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.012857 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.012816    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.013178 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.012925    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.013670 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.013651    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.014182 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.014166    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.030955 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.030932    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgzc8\" (UniqueName: \"kubernetes.io/projected/d43fce5b-ab9d-4750-bd9c-ae0c7e829d78-kube-api-access-lgzc8\") pod \"alertmanager-main-0\" (UID: \"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.108944 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.108909    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 12:46:24.450648 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.450568    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 12:46:24.455621 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:46:24.455591    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43fce5b_ab9d_4750_bd9c_ae0c7e829d78.slice/crio-22458ae1ab6f803d0f2013f0db3a4e2bc45523b6e832a67ce6a2cbaca5359284 WatchSource:0}: Error finding container 22458ae1ab6f803d0f2013f0db3a4e2bc45523b6e832a67ce6a2cbaca5359284: Status 404 returned error can't find the container with id 22458ae1ab6f803d0f2013f0db3a4e2bc45523b6e832a67ce6a2cbaca5359284
Oct 30 12:46:24.529098 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.529071    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c94b3e-9026-4fba-9788-03f2d981b53b" path="/var/lib/kubelet/pods/63c94b3e-9026-4fba-9788-03f2d981b53b/volumes"
Oct 30 12:46:24.555918 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.555885    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7d587d759d-h9f5b" podUID="79d36201-724a-4cea-be2b-2e23dbf45682" containerName="console" containerID="cri-o://69ce50314b5f38bc1c6bae178174c5cc37252fd6f32432c139505b2fd03b915c" gracePeriod=15
Oct 30 12:46:24.706410 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.706335    5893 generic.go:356] "Generic (PLEG): container finished" podID="79d36201-724a-4cea-be2b-2e23dbf45682" containerID="69ce50314b5f38bc1c6bae178174c5cc37252fd6f32432c139505b2fd03b915c" exitCode=0
Oct 30 12:46:24.706582 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.706410    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d587d759d-h9f5b" event={"ID":"79d36201-724a-4cea-be2b-2e23dbf45682","Type":"ContainerDied","Data":"69ce50314b5f38bc1c6bae178174c5cc37252fd6f32432c139505b2fd03b915c"}
Oct 30 12:46:24.707941 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.707913    5893 generic.go:356] "Generic (PLEG): container finished" podID="d43fce5b-ab9d-4750-bd9c-ae0c7e829d78" containerID="caef728b81b56c334c8187e478d001cd148c22517297e2ba59c17b2b83a4a3a0" exitCode=0
Oct 30 12:46:24.708035 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.707968    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78","Type":"ContainerDied","Data":"caef728b81b56c334c8187e478d001cd148c22517297e2ba59c17b2b83a4a3a0"}
Oct 30 12:46:24.708082 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.708033    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78","Type":"ContainerStarted","Data":"22458ae1ab6f803d0f2013f0db3a4e2bc45523b6e832a67ce6a2cbaca5359284"}
Oct 30 12:46:24.791252 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.791230    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:24.919117 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919094    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert\") pod \"79d36201-724a-4cea-be2b-2e23dbf45682\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") "
Oct 30 12:46:24.919255 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919155    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config\") pod \"79d36201-724a-4cea-be2b-2e23dbf45682\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") "
Oct 30 12:46:24.919255 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919194    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert\") pod \"79d36201-724a-4cea-be2b-2e23dbf45682\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") "
Oct 30 12:46:24.919379 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919256    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config\") pod \"79d36201-724a-4cea-be2b-2e23dbf45682\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") "
Oct 30 12:46:24.919379 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919318    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml\" (UniqueName: \"kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml\") pod \"79d36201-724a-4cea-be2b-2e23dbf45682\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") "
Oct 30 12:46:24.919379 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919360    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle\") pod \"79d36201-724a-4cea-be2b-2e23dbf45682\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") "
Oct 30 12:46:24.919527 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919401    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca\") pod \"79d36201-724a-4cea-be2b-2e23dbf45682\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") "
Oct 30 12:46:24.919527 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919429    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79d36201-724a-4cea-be2b-2e23dbf45682-tmp\" (UniqueName: \"kubernetes.io/empty-dir/79d36201-724a-4cea-be2b-2e23dbf45682-tmp\") pod \"79d36201-724a-4cea-be2b-2e23dbf45682\" (UID: \"79d36201-724a-4cea-be2b-2e23dbf45682\") "
Oct 30 12:46:24.919639 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919522    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert" pod "79d36201-724a-4cea-be2b-2e23dbf45682" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:24.919639 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919593    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config" pod "79d36201-724a-4cea-be2b-2e23dbf45682" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:24.919736 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919723    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79d36201-724a-4cea-be2b-2e23dbf45682-tmp" pod "79d36201-724a-4cea-be2b-2e23dbf45682" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:24.919838 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919820    5893 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79d36201-724a-4cea-be2b-2e23dbf45682-tmp\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:24.919897 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919844    5893 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-oauth-serving-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:24.919897 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919860    5893 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-console-config\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:24.919999 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.919971    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle" pod "79d36201-724a-4cea-be2b-2e23dbf45682" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:24.920051 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.920005    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca" pod "79d36201-724a-4cea-be2b-2e23dbf45682" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:24.922558 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.922491    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config" pod "79d36201-724a-4cea-be2b-2e23dbf45682" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:24.922615 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.922577    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert" pod "79d36201-724a-4cea-be2b-2e23dbf45682" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:24.922615 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:24.922606    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml" pod "79d36201-724a-4cea-be2b-2e23dbf45682" (UID: "79d36201-724a-4cea-be2b-2e23dbf45682"). InnerVolumeSpecName "kube-api-access-c6rml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:25.020684 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.020646    5893 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-serving-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:25.020684 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.020682    5893 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/79d36201-724a-4cea-be2b-2e23dbf45682-console-oauth-config\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:25.020803 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.020698    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6rml\" (UniqueName: \"kubernetes.io/projected/79d36201-724a-4cea-be2b-2e23dbf45682-kube-api-access-c6rml\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:25.020803 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.020747    5893 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-trusted-ca-bundle\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:25.020803 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.020762    5893 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/79d36201-724a-4cea-be2b-2e23dbf45682-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:25.718400 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.718366    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78","Type":"ContainerStarted","Data":"6cbe25f9eed3a6f1d547f70740df84274cdd44e441fb19f247d12814a0d2c7e4"}
Oct 30 12:46:25.718862 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.718443    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78","Type":"ContainerStarted","Data":"6ae1c6934acb5776ed5ad0d1ba22d73fcc82e0bd2f3eefcc2ba64025174e88c3"}
Oct 30 12:46:25.718862 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.718473    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78","Type":"ContainerStarted","Data":"54e01f9a8f772acf2b11bec7156306a7c97f175ada78636db8b323262ce1d2f4"}
Oct 30 12:46:25.718862 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.718499    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78","Type":"ContainerStarted","Data":"add3b48ec3b35bb84c18a47d5198c3a95a2f929ed33a4a58bc88553f6c460af4"}
Oct 30 12:46:25.718862 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.718529    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78","Type":"ContainerStarted","Data":"f0c88a35c852a89afd162322c562e0e598d13e9b42493ba12e37cc79849d9ca5"}
Oct 30 12:46:25.718862 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.718556    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d43fce5b-ab9d-4750-bd9c-ae0c7e829d78","Type":"ContainerStarted","Data":"40b549c750c7d8d6d11344ee4dd59843ca9dceaecbb60188458c6f2c7a3c4500"}
Oct 30 12:46:25.719730 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.719705    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d587d759d-h9f5b" event={"ID":"79d36201-724a-4cea-be2b-2e23dbf45682","Type":"ContainerDied","Data":"416452e21dafcd020d6cce7283ba268ad5811bcfa51e211b076baafe482a1e6e"}
Oct 30 12:46:25.719866 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.719732    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d587d759d-h9f5b"
Oct 30 12:46:25.719866 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.719761    5893 scope.go:122] "RemoveContainer" containerID="69ce50314b5f38bc1c6bae178174c5cc37252fd6f32432c139505b2fd03b915c"
Oct 30 12:46:25.808067 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.808015    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.7951784269999997 podStartE2EDuration="2.808002521s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.824094ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:25.806521873 +0000 UTC m=+27.389951567" watchObservedRunningTime="2026-10-30 12:46:25.808002521 +0000 UTC m=+27.391432213"
Oct 30 12:46:25.850761 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.850733    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d587d759d-h9f5b"]
Oct 30 12:46:25.852804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.852779    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d587d759d-h9f5b"]
Oct 30 12:46:25.943484 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.943458    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"]
Oct 30 12:46:25.943747 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.943737    5893 cpu_manager.go:414] "containerMap: removing container" podUID="79d36201-724a-4cea-be2b-2e23dbf45682" containerName="console"
Oct 30 12:46:25.943786 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.943749    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d36201-724a-4cea-be2b-2e23dbf45682" containerName="console"
Oct 30 12:46:25.943838 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.943831    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d36201-724a-4cea-be2b-2e23dbf45682" containerName="console"
Oct 30 12:46:25.949521 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.949499    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:25.966444 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.966421    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 12:46:25.966570 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.966533    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 12:46:25.966570 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.966544    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 12:46:25.966718 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.966702    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 12:46:25.966768 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.966737    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-x4gjk\""
Oct 30 12:46:25.976389 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:25.976327    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"]
Oct 30 12:46:26.030230 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.030199    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/74d25cd7-3283-4772-8267-c0f5d727633c-telemeter-client-tls\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.030451 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.030248    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74d25cd7-3283-4772-8267-c0f5d727633c-serving-certs-ca-bundle\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.030451 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.030298    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74d25cd7-3283-4772-8267-c0f5d727633c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.030451 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.030423    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/74d25cd7-3283-4772-8267-c0f5d727633c-federate-client-tls\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.030622 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.030470    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74d25cd7-3283-4772-8267-c0f5d727633c-metrics-client-ca\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.030622 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.030514    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/74d25cd7-3283-4772-8267-c0f5d727633c-secret-telemeter-client\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.030622 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.030568    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74d25cd7-3283-4772-8267-c0f5d727633c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.030622 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.030604    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66rdz\" (UniqueName: \"kubernetes.io/projected/74d25cd7-3283-4772-8267-c0f5d727633c-kube-api-access-66rdz\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.051910 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.051876    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 12:46:26.062494 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.062466    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 12:46:26.131255 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.131226    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74d25cd7-3283-4772-8267-c0f5d727633c-serving-certs-ca-bundle\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.131436 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.131268    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74d25cd7-3283-4772-8267-c0f5d727633c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.131436 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.131315    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/74d25cd7-3283-4772-8267-c0f5d727633c-federate-client-tls\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.131436 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.131333    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74d25cd7-3283-4772-8267-c0f5d727633c-metrics-client-ca\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.131436 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.131348    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/74d25cd7-3283-4772-8267-c0f5d727633c-secret-telemeter-client\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.131436 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.131361    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74d25cd7-3283-4772-8267-c0f5d727633c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.131436 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.131390    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66rdz\" (UniqueName: \"kubernetes.io/projected/74d25cd7-3283-4772-8267-c0f5d727633c-kube-api-access-66rdz\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.131713 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.131451    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/74d25cd7-3283-4772-8267-c0f5d727633c-telemeter-client-tls\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.132140 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.132111    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74d25cd7-3283-4772-8267-c0f5d727633c-serving-certs-ca-bundle\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.132247 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.132199    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74d25cd7-3283-4772-8267-c0f5d727633c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.132344 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.132251    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74d25cd7-3283-4772-8267-c0f5d727633c-metrics-client-ca\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.134121 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.134099    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/74d25cd7-3283-4772-8267-c0f5d727633c-telemeter-client-tls\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.134408 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.134389    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/74d25cd7-3283-4772-8267-c0f5d727633c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.134636 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.134616    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/74d25cd7-3283-4772-8267-c0f5d727633c-secret-telemeter-client\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.134703 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.134624    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/74d25cd7-3283-4772-8267-c0f5d727633c-federate-client-tls\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.152911 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.152886    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66rdz\" (UniqueName: \"kubernetes.io/projected/74d25cd7-3283-4772-8267-c0f5d727633c-kube-api-access-66rdz\") pod \"telemeter-client-7b454bf9f7-qlh67\" (UID: \"74d25cd7-3283-4772-8267-c0f5d727633c\") " pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.266211 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.266128    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"
Oct 30 12:46:26.513656 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.513630    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fd9xb"
Oct 30 12:46:26.531325 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.531231    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d36201-724a-4cea-be2b-2e23dbf45682" path="/var/lib/kubelet/pods/79d36201-724a-4cea-be2b-2e23dbf45682/volumes"
Oct 30 12:46:26.555296 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.555255    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:46:26.555595 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.555565    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="prometheus" containerID="cri-o://c82e4f116e13689a12702b09eb5c5c5d6ec9cbe795bb1f73d34d78e494d39c19" gracePeriod=600
Oct 30 12:46:26.555680 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.555578    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="kube-rbac-proxy-web" containerID="cri-o://8fa292dd75a6c79f7f212a0d09e2fc4ff77fc7369910fb5e9c63494a2db58959" gracePeriod=600
Oct 30 12:46:26.555680 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.555601    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="config-reloader" containerID="cri-o://401974b924fee06c62b687e370fa2a7b431adc89ab56e7709aee9ad6f2cae5e6" gracePeriod=600
Oct 30 12:46:26.555786 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.555592    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="kube-rbac-proxy" containerID="cri-o://e2e695ac1c14fb46d732d615cf850054de0e29b4d4a276fc10ab09ace1a2ca19" gracePeriod=600
Oct 30 12:46:26.555786 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.555568    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="kube-rbac-proxy-thanos" containerID="cri-o://b4d075a3d8c7215deb70dea8c7e479734eec1b0243e05b1eaf47517f6b4581d7" gracePeriod=600
Oct 30 12:46:26.556023 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.555586    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="thanos-sidecar" containerID="cri-o://b080e98d467a6afe0e2652e1b33796dbf26dccf93a6f10ab7221379bf076475d" gracePeriod=600
Oct 30 12:46:26.574761 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:26.574735    5893 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="c82e4f116e13689a12702b09eb5c5c5d6ec9cbe795bb1f73d34d78e494d39c19" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 12:46:26.574876 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:26.574782    5893 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="c82e4f116e13689a12702b09eb5c5c5d6ec9cbe795bb1f73d34d78e494d39c19" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 12:46:26.574876 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:26.574799    5893 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="c82e4f116e13689a12702b09eb5c5c5d6ec9cbe795bb1f73d34d78e494d39c19" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Oct 30 12:46:26.574876 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.574813    5893 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Oct 30 12:46:26.596646 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.596618    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b454bf9f7-qlh67"]
Oct 30 12:46:26.601749 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:46:26.601720    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d25cd7_3283_4772_8267_c0f5d727633c.slice/crio-131b018cddb1399711da486543b355f5a672b46218abc45138fb1c3cde560277 WatchSource:0}: Error finding container 131b018cddb1399711da486543b355f5a672b46218abc45138fb1c3cde560277: Status 404 returned error can't find the container with id 131b018cddb1399711da486543b355f5a672b46218abc45138fb1c3cde560277
Oct 30 12:46:26.663451 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.663426    5893 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:46:26.724584 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.724554    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67" event={"ID":"74d25cd7-3283-4772-8267-c0f5d727633c","Type":"ContainerStarted","Data":"131b018cddb1399711da486543b355f5a672b46218abc45138fb1c3cde560277"}
Oct 30 12:46:26.728217 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.728193    5893 generic.go:356] "Generic (PLEG): container finished" podID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerID="b4d075a3d8c7215deb70dea8c7e479734eec1b0243e05b1eaf47517f6b4581d7" exitCode=0
Oct 30 12:46:26.728217 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.728215    5893 generic.go:356] "Generic (PLEG): container finished" podID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerID="e2e695ac1c14fb46d732d615cf850054de0e29b4d4a276fc10ab09ace1a2ca19" exitCode=0
Oct 30 12:46:26.728217 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.728221    5893 generic.go:356] "Generic (PLEG): container finished" podID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerID="b080e98d467a6afe0e2652e1b33796dbf26dccf93a6f10ab7221379bf076475d" exitCode=0
Oct 30 12:46:26.728437 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.728227    5893 generic.go:356] "Generic (PLEG): container finished" podID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerID="401974b924fee06c62b687e370fa2a7b431adc89ab56e7709aee9ad6f2cae5e6" exitCode=0
Oct 30 12:46:26.728437 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.728232    5893 generic.go:356] "Generic (PLEG): container finished" podID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerID="c82e4f116e13689a12702b09eb5c5c5d6ec9cbe795bb1f73d34d78e494d39c19" exitCode=0
Oct 30 12:46:26.728437 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.728298    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerDied","Data":"b4d075a3d8c7215deb70dea8c7e479734eec1b0243e05b1eaf47517f6b4581d7"}
Oct 30 12:46:26.728437 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.728355    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerDied","Data":"e2e695ac1c14fb46d732d615cf850054de0e29b4d4a276fc10ab09ace1a2ca19"}
Oct 30 12:46:26.728437 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.728390    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerDied","Data":"b080e98d467a6afe0e2652e1b33796dbf26dccf93a6f10ab7221379bf076475d"}
Oct 30 12:46:26.728629 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.728445    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerDied","Data":"401974b924fee06c62b687e370fa2a7b431adc89ab56e7709aee9ad6f2cae5e6"}
Oct 30 12:46:26.728629 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:26.728476    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerDied","Data":"c82e4f116e13689a12702b09eb5c5c5d6ec9cbe795bb1f73d34d78e494d39c19"}
Oct 30 12:46:27.741712 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.741677    5893 generic.go:356] "Generic (PLEG): container finished" podID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerID="8fa292dd75a6c79f7f212a0d09e2fc4ff77fc7369910fb5e9c63494a2db58959" exitCode=0
Oct 30 12:46:27.742051 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.741785    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerDied","Data":"8fa292dd75a6c79f7f212a0d09e2fc4ff77fc7369910fb5e9c63494a2db58959"}
Oct 30 12:46:27.835124 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.835098    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959381    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-serving-certs-ca-bundle\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959437    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-web-config\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-web-config\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959472    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-grpc-tls\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959522    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-trusted-ca-bundle\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959555    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-thanos-prometheus-http-client-file\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959598    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-kubelet-serving-ca-bundle\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959633    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-kube-rbac-proxy\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959675    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-1\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959708    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-tls\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959741    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-metrics-client-certs\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959773    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-config-out\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959804    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-metrics-client-ca\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959848    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-config\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-config\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959880    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959934    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-2\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.959965    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.960003    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-0\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.960048    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-db\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.960079    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-tls-assets\" (UniqueName: \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-tls-assets\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.964345 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.960127    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-kube-api-access-789l8\" (UniqueName: \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-kube-api-access-789l8\") pod \"d457ae45-4f12-43bf-b6a5-634a29981a55\" (UID: \"d457ae45-4f12-43bf-b6a5-634a29981a55\") "
Oct 30 12:46:27.967983 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.967944    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-2" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:27.969337 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.969016    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-serving-certs-ca-bundle" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:27.969655 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.969626    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-trusted-ca-bundle" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:27.970052 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.970031    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-0" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:27.970576 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.970553    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-1" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:27.975475 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.975448    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-db" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:27.975738 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.975712    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-config-out" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:46:27.975947 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.975728    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-metrics-client-ca" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:27.976091 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.975966    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-thanos-prometheus-http-client-file" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:27.976240 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.976206    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-kubelet-serving-ca-bundle" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 12:46:27.978115 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.978004    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-tls-assets" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:27.978211 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.978167    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-tls" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:27.978420 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.978398    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-grpc-tls" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:27.978550 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.978459    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-config" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:27.978550 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.978531    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-kube-rbac-proxy" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:27.978686 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.978668    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-thanos-sidecar-tls" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:27.978806 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.978788    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-metrics-client-certs" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:27.980783 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.980720    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-kube-rbac-proxy-web" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:27.984297 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.982252    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-kube-api-access-789l8" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "kube-api-access-789l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:46:27.991589 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:27.991565    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-web-config" pod "d457ae45-4f12-43bf-b6a5-634a29981a55" (UID: "d457ae45-4f12-43bf-b6a5-634a29981a55"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:46:28.060970 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.060937    5893 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.060970 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.060971    5893 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.060970 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.060988    5893 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061003    5893 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-db\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061015    5893 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-tls-assets\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061027    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-789l8\" (UniqueName: \"kubernetes.io/projected/d457ae45-4f12-43bf-b6a5-634a29981a55-kube-api-access-789l8\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061038    5893 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061050    5893 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-web-config\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061063    5893 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-grpc-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061074    5893 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061086    5893 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061100    5893 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061112    5893 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-kube-rbac-proxy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061124    5893 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061137    5893 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061149    5893 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-metrics-client-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061161    5893 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d457ae45-4f12-43bf-b6a5-634a29981a55-config-out\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061174    5893 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d457ae45-4f12-43bf-b6a5-634a29981a55-configmap-metrics-client-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061188    5893 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-config\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.061250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.061202    5893 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d457ae45-4f12-43bf-b6a5-634a29981a55-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:46:28.753077 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.753033    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67" event={"ID":"74d25cd7-3283-4772-8267-c0f5d727633c","Type":"ContainerStarted","Data":"008106b6c79ed0b54829f4a223d00ac7b9f91ab756d2d71c3f7c7388e13c0868"}
Oct 30 12:46:28.753602 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.753133    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67" event={"ID":"74d25cd7-3283-4772-8267-c0f5d727633c","Type":"ContainerStarted","Data":"fa95e62918bb0c2ae26e7eaa26add117801392ff98572e111f53a898603c3106"}
Oct 30 12:46:28.753602 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.753171    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67" event={"ID":"74d25cd7-3283-4772-8267-c0f5d727633c","Type":"ContainerStarted","Data":"dcbfeb26c8a37fa85c81d95ead2262009de29833c5fbbade228b9475ee3e6465"}
Oct 30 12:46:28.756317 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.756266    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d457ae45-4f12-43bf-b6a5-634a29981a55","Type":"ContainerDied","Data":"80a9b43884d5c96da6c6eafdf9963cd816b6a79de6a12ec8eb1b7cf364f5ab63"}
Oct 30 12:46:28.756429 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.756344    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:28.756429 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.756377    5893 scope.go:122] "RemoveContainer" containerID="b4d075a3d8c7215deb70dea8c7e479734eec1b0243e05b1eaf47517f6b4581d7"
Oct 30 12:46:28.764182 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.764163    5893 scope.go:122] "RemoveContainer" containerID="e2e695ac1c14fb46d732d615cf850054de0e29b4d4a276fc10ab09ace1a2ca19"
Oct 30 12:46:28.771401 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.771380    5893 scope.go:122] "RemoveContainer" containerID="8fa292dd75a6c79f7f212a0d09e2fc4ff77fc7369910fb5e9c63494a2db58959"
Oct 30 12:46:28.778113 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.778097    5893 scope.go:122] "RemoveContainer" containerID="b080e98d467a6afe0e2652e1b33796dbf26dccf93a6f10ab7221379bf076475d"
Oct 30 12:46:28.785321 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.785304    5893 scope.go:122] "RemoveContainer" containerID="401974b924fee06c62b687e370fa2a7b431adc89ab56e7709aee9ad6f2cae5e6"
Oct 30 12:46:28.792133 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.792117    5893 scope.go:122] "RemoveContainer" containerID="c82e4f116e13689a12702b09eb5c5c5d6ec9cbe795bb1f73d34d78e494d39c19"
Oct 30 12:46:28.800100 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.800086    5893 scope.go:122] "RemoveContainer" containerID="af2b81871b51f98092e2ca59b7ad6cd7e58f4d8f62a86ba6b153408ca3a82817"
Oct 30 12:46:28.800220 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.800172    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7b454bf9f7-qlh67" podStartSLOduration=2.071577012 podStartE2EDuration="3.800158049s" totalImagesPullingTime="1.728581037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:28.799503036 +0000 UTC m=+30.382932728" watchObservedRunningTime="2026-10-30 12:46:28.800158049 +0000 UTC m=+30.383587732"
Oct 30 12:46:28.838595 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.838569    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:46:28.841074 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.841052    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:46:28.877073 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877032    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:46:28.877409 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877393    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="kube-rbac-proxy-thanos"
Oct 30 12:46:28.877456 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877410    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="kube-rbac-proxy-thanos"
Oct 30 12:46:28.877456 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877419    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="config-reloader"
Oct 30 12:46:28.877456 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877424    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="config-reloader"
Oct 30 12:46:28.877456 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877433    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="kube-rbac-proxy"
Oct 30 12:46:28.877456 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877438    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="kube-rbac-proxy"
Oct 30 12:46:28.877456 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877443    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="init-config-reloader"
Oct 30 12:46:28.877456 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877448    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="init-config-reloader"
Oct 30 12:46:28.877660 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877464    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="thanos-sidecar"
Oct 30 12:46:28.877660 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877468    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="thanos-sidecar"
Oct 30 12:46:28.877660 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877477    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="prometheus"
Oct 30 12:46:28.877660 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877480    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="prometheus"
Oct 30 12:46:28.877660 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877490    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="kube-rbac-proxy-web"
Oct 30 12:46:28.877660 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877495    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="kube-rbac-proxy-web"
Oct 30 12:46:28.877660 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877539    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="kube-rbac-proxy-web"
Oct 30 12:46:28.877660 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877548    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="kube-rbac-proxy-thanos"
Oct 30 12:46:28.877660 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877555    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="config-reloader"
Oct 30 12:46:28.877660 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877560    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="kube-rbac-proxy"
Oct 30 12:46:28.877660 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877564    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="init-config-reloader"
Oct 30 12:46:28.877660 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877571    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="thanos-sidecar"
Oct 30 12:46:28.877660 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.877577    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" containerName="prometheus"
Oct 30 12:46:28.882858 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.882840    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:28.899178 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.899149    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 12:46:28.899451 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.899432    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 12:46:28.899451 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.899443    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dcr46kt5lt1rp\""
Oct 30 12:46:28.899583 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.899441    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 12:46:28.899583 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.899469    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 12:46:28.899583 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.899501    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 12:46:28.899583 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.899511    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-gxtbq\""
Oct 30 12:46:28.899583 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.899520    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 12:46:28.899739 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.899655    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 12:46:28.899739 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.899707    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 12:46:28.899739 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.899716    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 12:46:28.908799 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.908780    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 12:46:28.908799 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.908791    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 12:46:28.908799 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.908792    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 12:46:28.908968 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.908864    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 12:46:28.913476 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.913459    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 12:46:28.923717 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:28.923689    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:46:29.069001 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.068918    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069001 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.068947    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmgw7\" (UniqueName: \"kubernetes.io/projected/c5a3e7ee-3d52-4005-aff1-f98799898c05-kube-api-access-bmgw7\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069001 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.068972    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069236 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069046    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c5a3e7ee-3d52-4005-aff1-f98799898c05-config-out\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069236 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069124    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069236 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069186    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069236 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069212    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069236 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069232    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069502 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069302    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069502 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069319    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069502 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069338    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-config\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069502 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069351    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c5a3e7ee-3d52-4005-aff1-f98799898c05-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069502 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069371    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069502 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069457    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069502 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069478    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069502 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069493    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069752 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069526    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069752 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069583    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069752 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069606    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-web-config\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.069752 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.069623    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170341 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170310    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170341 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170347    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c5a3e7ee-3d52-4005-aff1-f98799898c05-config-out\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170602 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170380    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170602 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170397    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170602 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170417    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170602 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170440    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170602 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170466    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170602 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170581    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170892 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170606    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-config\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170892 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170628    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c5a3e7ee-3d52-4005-aff1-f98799898c05-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170892 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170663    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170892 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170695    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170892 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170722    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170892 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170744    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170892 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170783    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170892 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170821    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170892 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170852    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-web-config\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.170892 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170880    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.171554 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170912    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.171554 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.170936    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmgw7\" (UniqueName: \"kubernetes.io/projected/c5a3e7ee-3d52-4005-aff1-f98799898c05-kube-api-access-bmgw7\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.171554 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.171105    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.171554 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.171263    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.171554 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.171268    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.171966 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.171944    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.173727 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.173337    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.174117 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.174070    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c5a3e7ee-3d52-4005-aff1-f98799898c05-config-out\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.174251 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.174233    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.174477 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.174420    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.174599 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.174573    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-web-config\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.174715 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.174680    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.175065 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.175042    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c5a3e7ee-3d52-4005-aff1-f98799898c05-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.175160 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.175068    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.175696 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.175633    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.176813 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.176788    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.176813 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.176800    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.176813 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.176806    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.177033 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.176948    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-config\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.177169 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.177151    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c5a3e7ee-3d52-4005-aff1-f98799898c05-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.177710 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.177693    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c5a3e7ee-3d52-4005-aff1-f98799898c05-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.192617 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.192574    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmgw7\" (UniqueName: \"kubernetes.io/projected/c5a3e7ee-3d52-4005-aff1-f98799898c05-kube-api-access-bmgw7\") pod \"prometheus-k8s-0\" (UID: \"c5a3e7ee-3d52-4005-aff1-f98799898c05\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.207739 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.207715    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:29.554146 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.554116    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 12:46:29.558602 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:46:29.558573    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a3e7ee_3d52_4005_aff1_f98799898c05.slice/crio-7e81ddbf20173fdb4a0bd36ad9d0caaf9e2edf7a7fb180276da5754057fdb431 WatchSource:0}: Error finding container 7e81ddbf20173fdb4a0bd36ad9d0caaf9e2edf7a7fb180276da5754057fdb431: Status 404 returned error can't find the container with id 7e81ddbf20173fdb4a0bd36ad9d0caaf9e2edf7a7fb180276da5754057fdb431
Oct 30 12:46:29.762609 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.762571    5893 generic.go:356] "Generic (PLEG): container finished" podID="c5a3e7ee-3d52-4005-aff1-f98799898c05" containerID="ddfaf7727b44013dd9714a01f768c45f28bf3c00d71cadf2681405ad3193c5fb" exitCode=0
Oct 30 12:46:29.762974 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.762667    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5a3e7ee-3d52-4005-aff1-f98799898c05","Type":"ContainerDied","Data":"ddfaf7727b44013dd9714a01f768c45f28bf3c00d71cadf2681405ad3193c5fb"}
Oct 30 12:46:29.762974 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:29.762750    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5a3e7ee-3d52-4005-aff1-f98799898c05","Type":"ContainerStarted","Data":"7e81ddbf20173fdb4a0bd36ad9d0caaf9e2edf7a7fb180276da5754057fdb431"}
Oct 30 12:46:30.528861 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:30.528829    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d457ae45-4f12-43bf-b6a5-634a29981a55" path="/var/lib/kubelet/pods/d457ae45-4f12-43bf-b6a5-634a29981a55/volumes"
Oct 30 12:46:30.769624 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:30.769598    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 12:46:30.769995 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:30.769972    5893 generic.go:356] "Generic (PLEG): container finished" podID="c5a3e7ee-3d52-4005-aff1-f98799898c05" containerID="7a19c078702ca0fdacf026771c9d629c99e44dcfcce0d71f726f4aca55188eb6" exitCode=2
Oct 30 12:46:30.770033 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:30.770007    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5a3e7ee-3d52-4005-aff1-f98799898c05","Type":"ContainerStarted","Data":"2f05ca71c27a7ca52b4920cd314730299c9106514b85f1597bc2bdf22f1966b1"}
Oct 30 12:46:30.770086 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:30.770060    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5a3e7ee-3d52-4005-aff1-f98799898c05","Type":"ContainerStarted","Data":"2116bd49192b61878f47b5043d123c4a8f1d36d84c76b58a52b9a241df6f2c55"}
Oct 30 12:46:30.770117 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:30.770087    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5a3e7ee-3d52-4005-aff1-f98799898c05","Type":"ContainerStarted","Data":"638ccc025e1e1ebd5fc14cdbffcc3397567a98ddd5096ceea626bcf64e0e7444"}
Oct 30 12:46:30.770117 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:30.770110    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5a3e7ee-3d52-4005-aff1-f98799898c05","Type":"ContainerStarted","Data":"e0cb52ed9fc9c1b1c5aff9b5ac06b0a7c3c80b2d4d19067fd99971c2cf9e0f93"}
Oct 30 12:46:30.770179 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:30.770132    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5a3e7ee-3d52-4005-aff1-f98799898c05","Type":"ContainerStarted","Data":"71835027d94e7dd151bde6e977f3014bb82d01e04f33a12b02ce4a7954d2ae38"}
Oct 30 12:46:30.774251 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:30.770492    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5a3e7ee-3d52-4005-aff1-f98799898c05","Type":"ContainerDied","Data":"7a19c078702ca0fdacf026771c9d629c99e44dcfcce0d71f726f4aca55188eb6"}
Oct 30 12:46:30.774251 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:30.770716    5893 scope.go:122] "RemoveContainer" containerID="7a19c078702ca0fdacf026771c9d629c99e44dcfcce0d71f726f4aca55188eb6"
Oct 30 12:46:31.778483 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:31.778458    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 12:46:31.778952 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:31.778929    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c5a3e7ee-3d52-4005-aff1-f98799898c05","Type":"ContainerStarted","Data":"0802abb6691e2937fcf2cf6485bc1952f95ae4f36b42846cec0b17682e3df0ae"}
Oct 30 12:46:31.863390 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:31.863342    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.847830654 podStartE2EDuration="3.863327176s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.496522ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:46:31.853508684 +0000 UTC m=+33.436938378" watchObservedRunningTime="2026-10-30 12:46:31.863327176 +0000 UTC m=+33.446756866"
Oct 30 12:46:32.606684 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:32.606654    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7qcht"
Oct 30 12:46:34.208839 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:46:34.208800    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:46:48.644348 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:46:48.644308    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:47:15.880092 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:47:15.880036    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:47:15.896604 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:47:15.896577    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:47:15.960108 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:47:15.960082    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 12:47:18.643725 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:47:18.643687    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:47:48.644832 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:47:48.644749    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:48:18.643762 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:48:18.643715    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:48:48.643825 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:48:48.643776    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:49:18.643806 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:49:18.643763    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:49:48.643691 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:49:48.643653    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:50:18.643776 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:50:18.643735    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:50:32.895495 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:32.895441    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-tc749"]
Oct 30 12:50:32.898922 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:32.898901    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tc749"
Oct 30 12:50:32.915163 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:32.915140    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-wgxg9\""
Oct 30 12:50:32.925480 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:32.925461    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 12:50:32.925480 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:32.925485    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 12:50:32.926214 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:32.926196    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tc749"]
Oct 30 12:50:33.036506 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:33.036470    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c332a20-17c9-44cb-b581-b01e4bb43bc8-bound-sa-token\") pod \"cert-manager-597b95cb84-tc749\" (UID: \"8c332a20-17c9-44cb-b581-b01e4bb43bc8\") " pod="cert-manager/cert-manager-597b95cb84-tc749"
Oct 30 12:50:33.036669 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:33.036594    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86s4\" (UniqueName: \"kubernetes.io/projected/8c332a20-17c9-44cb-b581-b01e4bb43bc8-kube-api-access-d86s4\") pod \"cert-manager-597b95cb84-tc749\" (UID: \"8c332a20-17c9-44cb-b581-b01e4bb43bc8\") " pod="cert-manager/cert-manager-597b95cb84-tc749"
Oct 30 12:50:33.137502 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:33.137471    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86s4\" (UniqueName: \"kubernetes.io/projected/8c332a20-17c9-44cb-b581-b01e4bb43bc8-kube-api-access-d86s4\") pod \"cert-manager-597b95cb84-tc749\" (UID: \"8c332a20-17c9-44cb-b581-b01e4bb43bc8\") " pod="cert-manager/cert-manager-597b95cb84-tc749"
Oct 30 12:50:33.137661 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:33.137524    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c332a20-17c9-44cb-b581-b01e4bb43bc8-bound-sa-token\") pod \"cert-manager-597b95cb84-tc749\" (UID: \"8c332a20-17c9-44cb-b581-b01e4bb43bc8\") " pod="cert-manager/cert-manager-597b95cb84-tc749"
Oct 30 12:50:33.159585 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:33.159524    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c332a20-17c9-44cb-b581-b01e4bb43bc8-bound-sa-token\") pod \"cert-manager-597b95cb84-tc749\" (UID: \"8c332a20-17c9-44cb-b581-b01e4bb43bc8\") " pod="cert-manager/cert-manager-597b95cb84-tc749"
Oct 30 12:50:33.159753 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:33.159731    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86s4\" (UniqueName: \"kubernetes.io/projected/8c332a20-17c9-44cb-b581-b01e4bb43bc8-kube-api-access-d86s4\") pod \"cert-manager-597b95cb84-tc749\" (UID: \"8c332a20-17c9-44cb-b581-b01e4bb43bc8\") " pod="cert-manager/cert-manager-597b95cb84-tc749"
Oct 30 12:50:33.226539 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:33.226509    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-tc749"
Oct 30 12:50:33.522924 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:33.522889    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-tc749"]
Oct 30 12:50:33.527838 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:50:33.527803    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c332a20_17c9_44cb_b581_b01e4bb43bc8.slice/crio-9bb76a470429b34b5a67b711065c88bbd9568980154fa7003f2a893663b41c15 WatchSource:0}: Error finding container 9bb76a470429b34b5a67b711065c88bbd9568980154fa7003f2a893663b41c15: Status 404 returned error can't find the container with id 9bb76a470429b34b5a67b711065c88bbd9568980154fa7003f2a893663b41c15
Oct 30 12:50:33.669948 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:33.669913    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tc749" event={"ID":"8c332a20-17c9-44cb-b581-b01e4bb43bc8","Type":"ContainerStarted","Data":"9bb76a470429b34b5a67b711065c88bbd9568980154fa7003f2a893663b41c15"}
Oct 30 12:50:36.684379 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:36.684268    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-tc749" event={"ID":"8c332a20-17c9-44cb-b581-b01e4bb43bc8","Type":"ContainerStarted","Data":"8914d142fba26410c3d2d612d7adbf8ea848509d968aae774522571596786cd7"}
Oct 30 12:50:36.722615 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:36.722572    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-tc749" podStartSLOduration=1.9027247680000001 podStartE2EDuration="4.722558864s" totalImagesPullingTime="2.819834096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:50:36.722077718 +0000 UTC m=+278.305507412" watchObservedRunningTime="2026-10-30 12:50:36.722558864 +0000 UTC m=+278.305988551"
Oct 30 12:50:48.644418 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:50:48.644343    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:50:54.933312 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:54.933264    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"]
Oct 30 12:50:54.987791 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:54.987758    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.008635 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.008600    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"]
Oct 30 12:50:55.017519 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.017486    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 12:50:55.017519 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.017516    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 12:50:55.020429 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.020411    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 12:50:55.020698 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.020680    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fmghg\""
Oct 30 12:50:55.020789 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.020707    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\""
Oct 30 12:50:55.020789 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.020749    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 12:50:55.029110 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.029082    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kck9n\" (UniqueName: \"kubernetes.io/projected/ad9763b3-b806-4dc2-a756-b3ac03f747c8-kube-api-access-kck9n\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.029210 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.029118    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad9763b3-b806-4dc2-a756-b3ac03f747c8-manifests\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.029210 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.029148    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad9763b3-b806-4dc2-a756-b3ac03f747c8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.029309 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.029267    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad9763b3-b806-4dc2-a756-b3ac03f747c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.029354 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.029323    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad9763b3-b806-4dc2-a756-b3ac03f747c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.029390 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.029373    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad9763b3-b806-4dc2-a756-b3ac03f747c8-tmp\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.130740 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.130712    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad9763b3-b806-4dc2-a756-b3ac03f747c8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.130921 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.130755    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad9763b3-b806-4dc2-a756-b3ac03f747c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.130921 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.130772    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad9763b3-b806-4dc2-a756-b3ac03f747c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.130921 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.130817    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad9763b3-b806-4dc2-a756-b3ac03f747c8-tmp\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.130921 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.130842    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kck9n\" (UniqueName: \"kubernetes.io/projected/ad9763b3-b806-4dc2-a756-b3ac03f747c8-kube-api-access-kck9n\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.130921 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.130861    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad9763b3-b806-4dc2-a756-b3ac03f747c8-manifests\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.131296 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.131250    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ad9763b3-b806-4dc2-a756-b3ac03f747c8-manifests\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.133371 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.133344    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ad9763b3-b806-4dc2-a756-b3ac03f747c8-tmp\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.133597 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.133574    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ad9763b3-b806-4dc2-a756-b3ac03f747c8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.133661 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.133602    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad9763b3-b806-4dc2-a756-b3ac03f747c8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.133661 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.133636    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ad9763b3-b806-4dc2-a756-b3ac03f747c8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.158451 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.158419    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kck9n\" (UniqueName: \"kubernetes.io/projected/ad9763b3-b806-4dc2-a756-b3ac03f747c8-kube-api-access-kck9n\") pod \"opendatahub-operator-controller-manager-7c4494f88b-zt6hf\" (UID: \"ad9763b3-b806-4dc2-a756-b3ac03f747c8\") " pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.319597 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.319567    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:50:55.693292 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.693199    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"]
Oct 30 12:50:55.696607 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:50:55.696578    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9763b3_b806_4dc2_a756_b3ac03f747c8.slice/crio-ea17f6dcd2e2a427a30d78b321c61384b9bf21a01b637a23ffbcdc84401aa832 WatchSource:0}: Error finding container ea17f6dcd2e2a427a30d78b321c61384b9bf21a01b637a23ffbcdc84401aa832: Status 404 returned error can't find the container with id ea17f6dcd2e2a427a30d78b321c61384b9bf21a01b637a23ffbcdc84401aa832
Oct 30 12:50:55.752783 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:55.752745    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf" event={"ID":"ad9763b3-b806-4dc2-a756-b3ac03f747c8","Type":"ContainerStarted","Data":"ea17f6dcd2e2a427a30d78b321c61384b9bf21a01b637a23ffbcdc84401aa832"}
Oct 30 12:50:58.542717 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:58.542695    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 12:50:58.545822 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:58.545801    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 12:50:58.555663 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:58.555641    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 12:50:58.557642 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:58.557623    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 12:50:58.767546 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:58.767507    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf" event={"ID":"ad9763b3-b806-4dc2-a756-b3ac03f747c8","Type":"ContainerStarted","Data":"ac3620a3fd7d6e39d3f7d3fd1583459f1337dfdc8c446f9d3436a7ad36986f84"}
Oct 30 12:50:59.773537 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:59.773501    5893 generic.go:356] "Generic (PLEG): container finished" podID="ad9763b3-b806-4dc2-a756-b3ac03f747c8" containerID="ac3620a3fd7d6e39d3f7d3fd1583459f1337dfdc8c446f9d3436a7ad36986f84" exitCode=0
Oct 30 12:50:59.773953 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:50:59.773586    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf" event={"ID":"ad9763b3-b806-4dc2-a756-b3ac03f747c8","Type":"ContainerDied","Data":"ac3620a3fd7d6e39d3f7d3fd1583459f1337dfdc8c446f9d3436a7ad36986f84"}
Oct 30 12:51:00.780197 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:00.780163    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf" event={"ID":"ad9763b3-b806-4dc2-a756-b3ac03f747c8","Type":"ContainerStarted","Data":"8711e207a52495809150f75f24095b4422d2903c72379116b2f186bee02232e2"}
Oct 30 12:51:00.780587 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:00.780289    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:51:00.842537 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:00.842489    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf" podStartSLOduration=3.232672214 podStartE2EDuration="6.842474378s" totalImagesPullingTime="3.103964539s" totalInitContainerRuntime="505.837625ms" isStatefulPod=false podCreationTimestamp="2026-10-30 12:50:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:00.840225817 +0000 UTC m=+302.423655509" watchObservedRunningTime="2026-10-30 12:51:00.842474378 +0000 UTC m=+302.425904070"
Oct 30 12:51:11.786232 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:11.786203    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c4494f88b-zt6hf"
Oct 30 12:51:12.372717 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.372680    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"]
Oct 30 12:51:12.376725 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.376698    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:12.402287 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.402251    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Oct 30 12:51:12.411814 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.411790    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Oct 30 12:51:12.413131 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.413046    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Oct 30 12:51:12.413131 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.413050    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Oct 30 12:51:12.413312 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.413141    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Oct 30 12:51:12.413724 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.413705    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"]
Oct 30 12:51:12.486839 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.486795    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq7sb\" (UniqueName: \"kubernetes.io/projected/0ad44910-27e2-498e-b721-1dcbddfbfc1d-kube-api-access-jq7sb\") pod \"lws-controller-manager-55d66777d-g8wdl\" (UID: \"0ad44910-27e2-498e-b721-1dcbddfbfc1d\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:12.487033 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.486881    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ad44910-27e2-498e-b721-1dcbddfbfc1d-metrics-cert\") pod \"lws-controller-manager-55d66777d-g8wdl\" (UID: \"0ad44910-27e2-498e-b721-1dcbddfbfc1d\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:12.487116 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.487024    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ad44910-27e2-498e-b721-1dcbddfbfc1d-cert\") pod \"lws-controller-manager-55d66777d-g8wdl\" (UID: \"0ad44910-27e2-498e-b721-1dcbddfbfc1d\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:12.487116 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.487070    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ad44910-27e2-498e-b721-1dcbddfbfc1d-manager-config\") pod \"lws-controller-manager-55d66777d-g8wdl\" (UID: \"0ad44910-27e2-498e-b721-1dcbddfbfc1d\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:12.502590 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.502561    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hw9dj\""
Oct 30 12:51:12.587771 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.587732    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ad44910-27e2-498e-b721-1dcbddfbfc1d-metrics-cert\") pod \"lws-controller-manager-55d66777d-g8wdl\" (UID: \"0ad44910-27e2-498e-b721-1dcbddfbfc1d\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:12.587771 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.587778    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ad44910-27e2-498e-b721-1dcbddfbfc1d-cert\") pod \"lws-controller-manager-55d66777d-g8wdl\" (UID: \"0ad44910-27e2-498e-b721-1dcbddfbfc1d\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:12.587985 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.587802    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ad44910-27e2-498e-b721-1dcbddfbfc1d-manager-config\") pod \"lws-controller-manager-55d66777d-g8wdl\" (UID: \"0ad44910-27e2-498e-b721-1dcbddfbfc1d\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:12.587985 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.587858    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq7sb\" (UniqueName: \"kubernetes.io/projected/0ad44910-27e2-498e-b721-1dcbddfbfc1d-kube-api-access-jq7sb\") pod \"lws-controller-manager-55d66777d-g8wdl\" (UID: \"0ad44910-27e2-498e-b721-1dcbddfbfc1d\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:12.588601 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.588575    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ad44910-27e2-498e-b721-1dcbddfbfc1d-manager-config\") pod \"lws-controller-manager-55d66777d-g8wdl\" (UID: \"0ad44910-27e2-498e-b721-1dcbddfbfc1d\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:12.590457 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.590433    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ad44910-27e2-498e-b721-1dcbddfbfc1d-metrics-cert\") pod \"lws-controller-manager-55d66777d-g8wdl\" (UID: \"0ad44910-27e2-498e-b721-1dcbddfbfc1d\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:12.590575 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.590519    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ad44910-27e2-498e-b721-1dcbddfbfc1d-cert\") pod \"lws-controller-manager-55d66777d-g8wdl\" (UID: \"0ad44910-27e2-498e-b721-1dcbddfbfc1d\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:12.609860 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.609833    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq7sb\" (UniqueName: \"kubernetes.io/projected/0ad44910-27e2-498e-b721-1dcbddfbfc1d-kube-api-access-jq7sb\") pod \"lws-controller-manager-55d66777d-g8wdl\" (UID: \"0ad44910-27e2-498e-b721-1dcbddfbfc1d\") " pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:12.688943 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:12.688865    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:13.032534 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:51:13.032506    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad44910_27e2_498e_b721_1dcbddfbfc1d.slice/crio-069c81fa7f2cd32a97ac27da8b1dcd96624ccf2a1d37c4a609c3e150e51fbfd3 WatchSource:0}: Error finding container 069c81fa7f2cd32a97ac27da8b1dcd96624ccf2a1d37c4a609c3e150e51fbfd3: Status 404 returned error can't find the container with id 069c81fa7f2cd32a97ac27da8b1dcd96624ccf2a1d37c4a609c3e150e51fbfd3
Oct 30 12:51:13.033034 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:13.033011    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"]
Oct 30 12:51:13.832592 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:13.832559    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl" event={"ID":"0ad44910-27e2-498e-b721-1dcbddfbfc1d","Type":"ContainerStarted","Data":"069c81fa7f2cd32a97ac27da8b1dcd96624ccf2a1d37c4a609c3e150e51fbfd3"}
Oct 30 12:51:15.843771 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:15.843739    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl" event={"ID":"0ad44910-27e2-498e-b721-1dcbddfbfc1d","Type":"ContainerStarted","Data":"e04e2b2e6ea821345f0ecc718a36d9b05d4bf3285403be042263dc7d82ed4e5a"}
Oct 30 12:51:15.844226 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:15.843919    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:15.886416 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:15.886360    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl" podStartSLOduration=1.592039531 podStartE2EDuration="3.886343654s" totalImagesPullingTime="2.294304123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:15.885855658 +0000 UTC m=+317.469285352" watchObservedRunningTime="2026-10-30 12:51:15.886343654 +0000 UTC m=+317.469773345"
Oct 30 12:51:18.644735 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:51:18.644697    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:51:26.850213 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:26.850186    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-55d66777d-g8wdl"
Oct 30 12:51:26.923854 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:26.923812    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr"]
Oct 30 12:51:26.927421 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:26.927400    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr"
Oct 30 12:51:26.945438 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:26.945411    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 12:51:26.945438 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:26.945424    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 12:51:26.945438 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:26.945432    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xgj4z\""
Oct 30 12:51:26.949903 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:26.949881    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr"]
Oct 30 12:51:27.010850 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.010802    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2vf2\" (UniqueName: \"kubernetes.io/projected/82007c50-3dfd-457b-9e0c-8d69f0481a9f-kube-api-access-f2vf2\") pod \"kube-auth-proxy-cbd4cb4f6-7pqlr\" (UID: \"82007c50-3dfd-457b-9e0c-8d69f0481a9f\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr"
Oct 30 12:51:27.011052 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.010873    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82007c50-3dfd-457b-9e0c-8d69f0481a9f-tmp\") pod \"kube-auth-proxy-cbd4cb4f6-7pqlr\" (UID: \"82007c50-3dfd-457b-9e0c-8d69f0481a9f\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr"
Oct 30 12:51:27.011173 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.011152    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82007c50-3dfd-457b-9e0c-8d69f0481a9f-tls-certs\") pod \"kube-auth-proxy-cbd4cb4f6-7pqlr\" (UID: \"82007c50-3dfd-457b-9e0c-8d69f0481a9f\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr"
Oct 30 12:51:27.030463 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.030441    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 12:51:27.030620 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.030607    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 12:51:27.112419 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.112323    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vf2\" (UniqueName: \"kubernetes.io/projected/82007c50-3dfd-457b-9e0c-8d69f0481a9f-kube-api-access-f2vf2\") pod \"kube-auth-proxy-cbd4cb4f6-7pqlr\" (UID: \"82007c50-3dfd-457b-9e0c-8d69f0481a9f\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr"
Oct 30 12:51:27.112419 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.112371    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82007c50-3dfd-457b-9e0c-8d69f0481a9f-tmp\") pod \"kube-auth-proxy-cbd4cb4f6-7pqlr\" (UID: \"82007c50-3dfd-457b-9e0c-8d69f0481a9f\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr"
Oct 30 12:51:27.112419 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.112412    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82007c50-3dfd-457b-9e0c-8d69f0481a9f-tls-certs\") pod \"kube-auth-proxy-cbd4cb4f6-7pqlr\" (UID: \"82007c50-3dfd-457b-9e0c-8d69f0481a9f\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr"
Oct 30 12:51:27.114847 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.114810    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82007c50-3dfd-457b-9e0c-8d69f0481a9f-tmp\") pod \"kube-auth-proxy-cbd4cb4f6-7pqlr\" (UID: \"82007c50-3dfd-457b-9e0c-8d69f0481a9f\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr"
Oct 30 12:51:27.115047 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.115028    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82007c50-3dfd-457b-9e0c-8d69f0481a9f-tls-certs\") pod \"kube-auth-proxy-cbd4cb4f6-7pqlr\" (UID: \"82007c50-3dfd-457b-9e0c-8d69f0481a9f\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr"
Oct 30 12:51:27.133943 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.133917    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vf2\" (UniqueName: \"kubernetes.io/projected/82007c50-3dfd-457b-9e0c-8d69f0481a9f-kube-api-access-f2vf2\") pod \"kube-auth-proxy-cbd4cb4f6-7pqlr\" (UID: \"82007c50-3dfd-457b-9e0c-8d69f0481a9f\") " pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr"
Oct 30 12:51:27.240298 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.240241    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr"
Oct 30 12:51:27.535695 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.535659    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr"]
Oct 30 12:51:27.540469 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:51:27.540443    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82007c50_3dfd_457b_9e0c_8d69f0481a9f.slice/crio-a1e2d4233e2fe740e895df7fe0a06f9d1903edc45e8335539cfad48a4b555148 WatchSource:0}: Error finding container a1e2d4233e2fe740e895df7fe0a06f9d1903edc45e8335539cfad48a4b555148: Status 404 returned error can't find the container with id a1e2d4233e2fe740e895df7fe0a06f9d1903edc45e8335539cfad48a4b555148
Oct 30 12:51:27.542361 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.542345    5893 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:51:27.895912 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:27.895834    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr" event={"ID":"82007c50-3dfd-457b-9e0c-8d69f0481a9f","Type":"ContainerStarted","Data":"a1e2d4233e2fe740e895df7fe0a06f9d1903edc45e8335539cfad48a4b555148"}
Oct 30 12:51:30.908888 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:30.908848    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr" event={"ID":"82007c50-3dfd-457b-9e0c-8d69f0481a9f","Type":"ContainerStarted","Data":"5fc766dbe18042dfa535e375ec83951ac1c5dcb87d82e89dd67a70e902525a18"}
Oct 30 12:51:30.951090 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:51:30.951032    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-cbd4cb4f6-7pqlr" podStartSLOduration=2.604260879 podStartE2EDuration="4.951013296s" totalImagesPullingTime="2.346752417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:51:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:51:30.95039587 +0000 UTC m=+332.533825584" watchObservedRunningTime="2026-10-30 12:51:30.951013296 +0000 UTC m=+332.534442990"
Oct 30 12:51:48.643755 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:51:48.643720    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:52:18.644760 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:52:18.644724    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:52:20.866235 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:20.866201    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-gpsxn"]
Oct 30 12:52:20.870964 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:20.870945    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn"
Oct 30 12:52:20.892299 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:20.892260    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-gpsxn"]
Oct 30 12:52:20.893010 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:20.892981    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd2zj\" (UniqueName: \"kubernetes.io/projected/3b1c3888-ac99-45cd-b338-c0b8418ccec2-kube-api-access-qd2zj\") pod \"odh-model-controller-f87fcf56c-gpsxn\" (UID: \"3b1c3888-ac99-45cd-b338-c0b8418ccec2\") " pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn"
Oct 30 12:52:20.893116 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:20.893053    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b1c3888-ac99-45cd-b338-c0b8418ccec2-cert\") pod \"odh-model-controller-f87fcf56c-gpsxn\" (UID: \"3b1c3888-ac99-45cd-b338-c0b8418ccec2\") " pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn"
Oct 30 12:52:20.893116 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:20.893101    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1c3888-ac99-45cd-b338-c0b8418ccec2-metrics-certs\") pod \"odh-model-controller-f87fcf56c-gpsxn\" (UID: \"3b1c3888-ac99-45cd-b338-c0b8418ccec2\") " pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn"
Oct 30 12:52:20.934397 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:20.934370    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 12:52:20.934683 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:20.934633    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 12:52:20.934944 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:20.934923    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-sxkvg\""
Oct 30 12:52:20.993919 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:20.993891    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1c3888-ac99-45cd-b338-c0b8418ccec2-metrics-certs\") pod \"odh-model-controller-f87fcf56c-gpsxn\" (UID: \"3b1c3888-ac99-45cd-b338-c0b8418ccec2\") " pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn"
Oct 30 12:52:20.994064 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:20.993931    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd2zj\" (UniqueName: \"kubernetes.io/projected/3b1c3888-ac99-45cd-b338-c0b8418ccec2-kube-api-access-qd2zj\") pod \"odh-model-controller-f87fcf56c-gpsxn\" (UID: \"3b1c3888-ac99-45cd-b338-c0b8418ccec2\") " pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn"
Oct 30 12:52:20.994064 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:20.993982    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b1c3888-ac99-45cd-b338-c0b8418ccec2-cert\") pod \"odh-model-controller-f87fcf56c-gpsxn\" (UID: \"3b1c3888-ac99-45cd-b338-c0b8418ccec2\") " pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn"
Oct 30 12:52:20.996514 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:20.996488    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b1c3888-ac99-45cd-b338-c0b8418ccec2-metrics-certs\") pod \"odh-model-controller-f87fcf56c-gpsxn\" (UID: \"3b1c3888-ac99-45cd-b338-c0b8418ccec2\") " pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn"
Oct 30 12:52:20.996514 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:20.996511    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b1c3888-ac99-45cd-b338-c0b8418ccec2-cert\") pod \"odh-model-controller-f87fcf56c-gpsxn\" (UID: \"3b1c3888-ac99-45cd-b338-c0b8418ccec2\") " pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn"
Oct 30 12:52:21.049937 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:21.049911    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd2zj\" (UniqueName: \"kubernetes.io/projected/3b1c3888-ac99-45cd-b338-c0b8418ccec2-kube-api-access-qd2zj\") pod \"odh-model-controller-f87fcf56c-gpsxn\" (UID: \"3b1c3888-ac99-45cd-b338-c0b8418ccec2\") " pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn"
Oct 30 12:52:21.183588 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:21.183504    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn"
Oct 30 12:52:21.480264 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:21.480228    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-gpsxn"]
Oct 30 12:52:21.484633 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:52:21.484606    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b1c3888_ac99_45cd_b338_c0b8418ccec2.slice/crio-f71a5df341ad2ccb32afc64a03529508a792364df2eb89a2dd219d679d6eab6d WatchSource:0}: Error finding container f71a5df341ad2ccb32afc64a03529508a792364df2eb89a2dd219d679d6eab6d: Status 404 returned error can't find the container with id f71a5df341ad2ccb32afc64a03529508a792364df2eb89a2dd219d679d6eab6d
Oct 30 12:52:22.121413 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:22.121378    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn" event={"ID":"3b1c3888-ac99-45cd-b338-c0b8418ccec2","Type":"ContainerStarted","Data":"f71a5df341ad2ccb32afc64a03529508a792364df2eb89a2dd219d679d6eab6d"}
Oct 30 12:52:24.132883 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:24.132787    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn" event={"ID":"3b1c3888-ac99-45cd-b338-c0b8418ccec2","Type":"ContainerStarted","Data":"c0c6dd895c335926a7aee82658ec95c4a83df553d86e51bd721f0596710ee0b9"}
Oct 30 12:52:24.133342 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:24.132898    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn"
Oct 30 12:52:24.180601 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:24.180553    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn" podStartSLOduration=1.811722041 podStartE2EDuration="4.180529594s" totalImagesPullingTime="2.368807553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:52:24.177768048 +0000 UTC m=+385.761197741" watchObservedRunningTime="2026-10-30 12:52:24.180529594 +0000 UTC m=+385.763959286"
Oct 30 12:52:35.138935 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:52:35.138904    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-gpsxn"
Oct 30 12:52:48.644390 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:52:48.644355    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:53:18.644123 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:53:18.644081    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:53:31.497675 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:31.497644    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"]
Oct 30 12:53:31.500560 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:31.500542    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:31.516719 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:31.516697    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-k8kf6\""
Oct 30 12:53:31.516873 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:31.516764    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 12:53:31.526697 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:31.526675    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 12:53:31.527700 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:31.527681    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"]
Oct 30 12:53:31.631573 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:31.631542    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40777d84-06e8-4105-8bc1-9a351c766ac7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-fvdnn\" (UID: \"40777d84-06e8-4105-8bc1-9a351c766ac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:31.631573 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:31.631575    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l8t7\" (UniqueName: \"kubernetes.io/projected/40777d84-06e8-4105-8bc1-9a351c766ac7-kube-api-access-4l8t7\") pod \"kuadrant-operator-controller-manager-6b6fb56846-fvdnn\" (UID: \"40777d84-06e8-4105-8bc1-9a351c766ac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:31.735517 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:31.735483    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40777d84-06e8-4105-8bc1-9a351c766ac7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-fvdnn\" (UID: \"40777d84-06e8-4105-8bc1-9a351c766ac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:31.735517 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:31.735515    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l8t7\" (UniqueName: \"kubernetes.io/projected/40777d84-06e8-4105-8bc1-9a351c766ac7-kube-api-access-4l8t7\") pod \"kuadrant-operator-controller-manager-6b6fb56846-fvdnn\" (UID: \"40777d84-06e8-4105-8bc1-9a351c766ac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:31.735936 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:31.735912    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40777d84-06e8-4105-8bc1-9a351c766ac7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-fvdnn\" (UID: \"40777d84-06e8-4105-8bc1-9a351c766ac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:31.757514 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:31.757459    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l8t7\" (UniqueName: \"kubernetes.io/projected/40777d84-06e8-4105-8bc1-9a351c766ac7-kube-api-access-4l8t7\") pod \"kuadrant-operator-controller-manager-6b6fb56846-fvdnn\" (UID: \"40777d84-06e8-4105-8bc1-9a351c766ac7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:31.813829 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:31.813794    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:32.146193 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:32.146109    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"]
Oct 30 12:53:32.151734 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:53:32.151707    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40777d84_06e8_4105_8bc1_9a351c766ac7.slice/crio-df7c13b93fdb44e62e0d757e93ed45ea772e856b7363b36affb055dd2d4a213a WatchSource:0}: Error finding container df7c13b93fdb44e62e0d757e93ed45ea772e856b7363b36affb055dd2d4a213a: Status 404 returned error can't find the container with id df7c13b93fdb44e62e0d757e93ed45ea772e856b7363b36affb055dd2d4a213a
Oct 30 12:53:32.414982 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:32.414895    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn" event={"ID":"40777d84-06e8-4105-8bc1-9a351c766ac7","Type":"ContainerStarted","Data":"df7c13b93fdb44e62e0d757e93ed45ea772e856b7363b36affb055dd2d4a213a"}
Oct 30 12:53:36.436104 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:36.436012    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn" event={"ID":"40777d84-06e8-4105-8bc1-9a351c766ac7","Type":"ContainerStarted","Data":"b131df4a2ebec9dcab2f19d27796d63519791a51514fb696f6d75fa540c33d8a"}
Oct 30 12:53:36.436549 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:36.436148    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:36.495837 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:36.495778    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn" podStartSLOduration=1.5403607400000001 podStartE2EDuration="5.495762529s" totalImagesPullingTime="3.955401789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:36.485244759 +0000 UTC m=+458.068674468" watchObservedRunningTime="2026-10-30 12:53:36.495762529 +0000 UTC m=+458.079192221"
Oct 30 12:53:47.443089 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:47.442996    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:48.644154 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:53:48.644117    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:53:51.301441 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.301404    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"]
Oct 30 12:53:51.302003 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.301679    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn" podUID="40777d84-06e8-4105-8bc1-9a351c766ac7" containerName="manager" containerID="cri-o://b131df4a2ebec9dcab2f19d27796d63519791a51514fb696f6d75fa540c33d8a" gracePeriod=2
Oct 30 12:53:51.302757 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.302465    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"]
Oct 30 12:53:51.314108 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:53:51.313354    5893 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-fvdnn\" is forbidden: User \"system:node:ip-10-0-141-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-232.ec2.internal' and this object" podUID="40777d84-06e8-4105-8bc1-9a351c766ac7" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:51.319133 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.319105    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"]
Oct 30 12:53:51.319615 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.319596    5893 cpu_manager.go:414] "containerMap: removing container" podUID="40777d84-06e8-4105-8bc1-9a351c766ac7" containerName="manager"
Oct 30 12:53:51.319785 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.319617    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40777d84-06e8-4105-8bc1-9a351c766ac7" containerName="manager"
Oct 30 12:53:51.319785 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.319721    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="40777d84-06e8-4105-8bc1-9a351c766ac7" containerName="manager"
Oct 30 12:53:51.322765 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.322745    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"
Oct 30 12:53:51.346425 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.346395    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"]
Oct 30 12:53:51.382408 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:53:51.382363    5893 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-fvdnn\" is forbidden: User \"system:node:ip-10-0-141-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-232.ec2.internal' and this object" podUID="40777d84-06e8-4105-8bc1-9a351c766ac7" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:51.430995 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.430961    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vqf2\" (UniqueName: \"kubernetes.io/projected/a0a13e8d-3947-425a-a5ec-869956621358-kube-api-access-5vqf2\") pod \"kuadrant-operator-controller-manager-6b6fb56846-t2lvg\" (UID: \"a0a13e8d-3947-425a-a5ec-869956621358\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"
Oct 30 12:53:51.431165 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.431052    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0a13e8d-3947-425a-a5ec-869956621358-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-t2lvg\" (UID: \"a0a13e8d-3947-425a-a5ec-869956621358\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"
Oct 30 12:53:51.502093 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.502061    5893 generic.go:356] "Generic (PLEG): container finished" podID="40777d84-06e8-4105-8bc1-9a351c766ac7" containerID="b131df4a2ebec9dcab2f19d27796d63519791a51514fb696f6d75fa540c33d8a" exitCode=0
Oct 30 12:53:51.531794 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.531765    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vqf2\" (UniqueName: \"kubernetes.io/projected/a0a13e8d-3947-425a-a5ec-869956621358-kube-api-access-5vqf2\") pod \"kuadrant-operator-controller-manager-6b6fb56846-t2lvg\" (UID: \"a0a13e8d-3947-425a-a5ec-869956621358\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"
Oct 30 12:53:51.531958 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.531832    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0a13e8d-3947-425a-a5ec-869956621358-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-t2lvg\" (UID: \"a0a13e8d-3947-425a-a5ec-869956621358\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"
Oct 30 12:53:51.532180 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.532159    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0a13e8d-3947-425a-a5ec-869956621358-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b6fb56846-t2lvg\" (UID: \"a0a13e8d-3947-425a-a5ec-869956621358\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"
Oct 30 12:53:51.553619 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.553557    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vqf2\" (UniqueName: \"kubernetes.io/projected/a0a13e8d-3947-425a-a5ec-869956621358-kube-api-access-5vqf2\") pod \"kuadrant-operator-controller-manager-6b6fb56846-t2lvg\" (UID: \"a0a13e8d-3947-425a-a5ec-869956621358\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"
Oct 30 12:53:51.554522 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.554505    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:51.565982 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:53:51.565949    5893 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-fvdnn\" is forbidden: User \"system:node:ip-10-0-141-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-232.ec2.internal' and this object" podUID="40777d84-06e8-4105-8bc1-9a351c766ac7" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:51.632725 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.632698    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40777d84-06e8-4105-8bc1-9a351c766ac7-kube-api-access-4l8t7\" (UniqueName: \"kubernetes.io/projected/40777d84-06e8-4105-8bc1-9a351c766ac7-kube-api-access-4l8t7\") pod \"40777d84-06e8-4105-8bc1-9a351c766ac7\" (UID: \"40777d84-06e8-4105-8bc1-9a351c766ac7\") "
Oct 30 12:53:51.632862 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.632743    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40777d84-06e8-4105-8bc1-9a351c766ac7-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40777d84-06e8-4105-8bc1-9a351c766ac7-extensions-socket-volume\") pod \"40777d84-06e8-4105-8bc1-9a351c766ac7\" (UID: \"40777d84-06e8-4105-8bc1-9a351c766ac7\") "
Oct 30 12:53:51.633303 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.633249    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40777d84-06e8-4105-8bc1-9a351c766ac7-extensions-socket-volume" pod "40777d84-06e8-4105-8bc1-9a351c766ac7" (UID: "40777d84-06e8-4105-8bc1-9a351c766ac7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:53:51.634950 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.634930    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40777d84-06e8-4105-8bc1-9a351c766ac7-kube-api-access-4l8t7" pod "40777d84-06e8-4105-8bc1-9a351c766ac7" (UID: "40777d84-06e8-4105-8bc1-9a351c766ac7"). InnerVolumeSpecName "kube-api-access-4l8t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:53:51.686134 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.686106    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"
Oct 30 12:53:51.734838 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.734540    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l8t7\" (UniqueName: \"kubernetes.io/projected/40777d84-06e8-4105-8bc1-9a351c766ac7-kube-api-access-4l8t7\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:53:51.734838 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:51.734571    5893 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/40777d84-06e8-4105-8bc1-9a351c766ac7-extensions-socket-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:53:52.032902 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:52.032871    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"]
Oct 30 12:53:52.038116 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:53:52.038088    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a13e8d_3947_425a_a5ec_869956621358.slice/crio-68ffb3ce23082e0bd695b40669649788d57f5d222f6fa80806dd9fb0ae32c08b WatchSource:0}: Error finding container 68ffb3ce23082e0bd695b40669649788d57f5d222f6fa80806dd9fb0ae32c08b: Status 404 returned error can't find the container with id 68ffb3ce23082e0bd695b40669649788d57f5d222f6fa80806dd9fb0ae32c08b
Oct 30 12:53:52.508321 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:52.508284    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg" event={"ID":"a0a13e8d-3947-425a-a5ec-869956621358","Type":"ContainerStarted","Data":"914fe2b1f2414f6684c6c090c1d076dd63044c7030aa182fa6c36d8c5494b2ac"}
Oct 30 12:53:52.508774 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:52.508347    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg" event={"ID":"a0a13e8d-3947-425a-a5ec-869956621358","Type":"ContainerStarted","Data":"68ffb3ce23082e0bd695b40669649788d57f5d222f6fa80806dd9fb0ae32c08b"}
Oct 30 12:53:52.508774 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:52.508370    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"
Oct 30 12:53:52.509673 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:52.509655    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:52.509673 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:52.509664    5893 scope.go:122] "RemoveContainer" containerID="b131df4a2ebec9dcab2f19d27796d63519791a51514fb696f6d75fa540c33d8a"
Oct 30 12:53:52.519999 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:53:52.519978    5893 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b6fb56846-fvdnn\" is forbidden: User \"system:node:ip-10-0-141-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-232.ec2.internal' and this object" podUID="40777d84-06e8-4105-8bc1-9a351c766ac7" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-fvdnn"
Oct 30 12:53:52.528109 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:52.528090    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40777d84-06e8-4105-8bc1-9a351c766ac7" path="/var/lib/kubelet/pods/40777d84-06e8-4105-8bc1-9a351c766ac7/volumes"
Oct 30 12:53:52.582520 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:53:52.582472    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg" podStartSLOduration=1.582459201 podStartE2EDuration="1.582459201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:53:52.571952713 +0000 UTC m=+474.155382408" watchObservedRunningTime="2026-10-30 12:53:52.582459201 +0000 UTC m=+474.165888894"
Oct 30 12:54:03.519045 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:03.519016    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"
Oct 30 12:54:05.932719 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:05.932685    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"]
Oct 30 12:54:05.933188 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:05.932945    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg" podUID="a0a13e8d-3947-425a-a5ec-869956621358" containerName="manager" containerID="cri-o://914fe2b1f2414f6684c6c090c1d076dd63044c7030aa182fa6c36d8c5494b2ac" gracePeriod=10
Oct 30 12:54:06.172968 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.172943    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"
Oct 30 12:54:06.203325 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.203227    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"]
Oct 30 12:54:06.203663 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.203644    5893 cpu_manager.go:414] "containerMap: removing container" podUID="a0a13e8d-3947-425a-a5ec-869956621358" containerName="manager"
Oct 30 12:54:06.203663 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.203660    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a13e8d-3947-425a-a5ec-869956621358" containerName="manager"
Oct 30 12:54:06.203846 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.203741    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a13e8d-3947-425a-a5ec-869956621358" containerName="manager"
Oct 30 12:54:06.205549 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.205533    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"
Oct 30 12:54:06.236783 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.236747    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"]
Oct 30 12:54:06.265883 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.265850    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a13e8d-3947-425a-a5ec-869956621358-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0a13e8d-3947-425a-a5ec-869956621358-extensions-socket-volume\") pod \"a0a13e8d-3947-425a-a5ec-869956621358\" (UID: \"a0a13e8d-3947-425a-a5ec-869956621358\") "
Oct 30 12:54:06.266073 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.265937    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a13e8d-3947-425a-a5ec-869956621358-kube-api-access-5vqf2\" (UniqueName: \"kubernetes.io/projected/a0a13e8d-3947-425a-a5ec-869956621358-kube-api-access-5vqf2\") pod \"a0a13e8d-3947-425a-a5ec-869956621358\" (UID: \"a0a13e8d-3947-425a-a5ec-869956621358\") "
Oct 30 12:54:06.266073 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.266055    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk46x\" (UniqueName: \"kubernetes.io/projected/520c958e-79f7-4563-af5d-3f313563da44-kube-api-access-hk46x\") pod \"kuadrant-operator-controller-manager-57bb9bfff-zq9bh\" (UID: \"520c958e-79f7-4563-af5d-3f313563da44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"
Oct 30 12:54:06.266203 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.266184    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/520c958e-79f7-4563-af5d-3f313563da44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-zq9bh\" (UID: \"520c958e-79f7-4563-af5d-3f313563da44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"
Oct 30 12:54:06.266335 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.266314    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a13e8d-3947-425a-a5ec-869956621358-extensions-socket-volume" pod "a0a13e8d-3947-425a-a5ec-869956621358" (UID: "a0a13e8d-3947-425a-a5ec-869956621358"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 12:54:06.268089 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.268071    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a13e8d-3947-425a-a5ec-869956621358-kube-api-access-5vqf2" pod "a0a13e8d-3947-425a-a5ec-869956621358" (UID: "a0a13e8d-3947-425a-a5ec-869956621358"). InnerVolumeSpecName "kube-api-access-5vqf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:54:06.367232 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.367189    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/520c958e-79f7-4563-af5d-3f313563da44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-zq9bh\" (UID: \"520c958e-79f7-4563-af5d-3f313563da44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"
Oct 30 12:54:06.367435 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.367255    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk46x\" (UniqueName: \"kubernetes.io/projected/520c958e-79f7-4563-af5d-3f313563da44-kube-api-access-hk46x\") pod \"kuadrant-operator-controller-manager-57bb9bfff-zq9bh\" (UID: \"520c958e-79f7-4563-af5d-3f313563da44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"
Oct 30 12:54:06.367435 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.367315    5893 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0a13e8d-3947-425a-a5ec-869956621358-extensions-socket-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:54:06.367435 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.367329    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vqf2\" (UniqueName: \"kubernetes.io/projected/a0a13e8d-3947-425a-a5ec-869956621358-kube-api-access-5vqf2\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:54:06.367620 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.367600    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/520c958e-79f7-4563-af5d-3f313563da44-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-zq9bh\" (UID: \"520c958e-79f7-4563-af5d-3f313563da44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"
Oct 30 12:54:06.400715 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.400680    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk46x\" (UniqueName: \"kubernetes.io/projected/520c958e-79f7-4563-af5d-3f313563da44-kube-api-access-hk46x\") pod \"kuadrant-operator-controller-manager-57bb9bfff-zq9bh\" (UID: \"520c958e-79f7-4563-af5d-3f313563da44\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"
Oct 30 12:54:06.517400 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.517370    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"
Oct 30 12:54:06.574466 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.574428    5893 generic.go:356] "Generic (PLEG): container finished" podID="a0a13e8d-3947-425a-a5ec-869956621358" containerID="914fe2b1f2414f6684c6c090c1d076dd63044c7030aa182fa6c36d8c5494b2ac" exitCode=0
Oct 30 12:54:06.574589 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.574513    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg" event={"ID":"a0a13e8d-3947-425a-a5ec-869956621358","Type":"ContainerDied","Data":"914fe2b1f2414f6684c6c090c1d076dd63044c7030aa182fa6c36d8c5494b2ac"}
Oct 30 12:54:06.574589 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.574566    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg" event={"ID":"a0a13e8d-3947-425a-a5ec-869956621358","Type":"ContainerDied","Data":"68ffb3ce23082e0bd695b40669649788d57f5d222f6fa80806dd9fb0ae32c08b"}
Oct 30 12:54:06.574700 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.574598    5893 scope.go:122] "RemoveContainer" containerID="914fe2b1f2414f6684c6c090c1d076dd63044c7030aa182fa6c36d8c5494b2ac"
Oct 30 12:54:06.574786 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.574768    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"
Oct 30 12:54:06.589391 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.589254    5893 scope.go:122] "RemoveContainer" containerID="914fe2b1f2414f6684c6c090c1d076dd63044c7030aa182fa6c36d8c5494b2ac"
Oct 30 12:54:06.589642 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:54:06.589619    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"914fe2b1f2414f6684c6c090c1d076dd63044c7030aa182fa6c36d8c5494b2ac\": container with ID starting with 914fe2b1f2414f6684c6c090c1d076dd63044c7030aa182fa6c36d8c5494b2ac not found: ID does not exist" containerID="914fe2b1f2414f6684c6c090c1d076dd63044c7030aa182fa6c36d8c5494b2ac"
Oct 30 12:54:06.589706 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.589656    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"914fe2b1f2414f6684c6c090c1d076dd63044c7030aa182fa6c36d8c5494b2ac"} err="failed to get container status \"914fe2b1f2414f6684c6c090c1d076dd63044c7030aa182fa6c36d8c5494b2ac\": rpc error: code = NotFound desc = could not find container \"914fe2b1f2414f6684c6c090c1d076dd63044c7030aa182fa6c36d8c5494b2ac\": container with ID starting with 914fe2b1f2414f6684c6c090c1d076dd63044c7030aa182fa6c36d8c5494b2ac not found: ID does not exist"
Oct 30 12:54:06.620990 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.620962    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"]
Oct 30 12:54:06.634105 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.634078    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b6fb56846-t2lvg"]
Oct 30 12:54:06.829154 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:06.829068    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"]
Oct 30 12:54:06.834446 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:54:06.834408    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520c958e_79f7_4563_af5d_3f313563da44.slice/crio-2f9546283bb14118565cf439a0a4fe0c6987c4729635ea776424d2058a6f4346 WatchSource:0}: Error finding container 2f9546283bb14118565cf439a0a4fe0c6987c4729635ea776424d2058a6f4346: Status 404 returned error can't find the container with id 2f9546283bb14118565cf439a0a4fe0c6987c4729635ea776424d2058a6f4346
Oct 30 12:54:07.581800 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:07.581759    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh" event={"ID":"520c958e-79f7-4563-af5d-3f313563da44","Type":"ContainerStarted","Data":"621b19b9366cb91baa2bdb300ab8efefb67be776578e4449e8bd4a9f92083850"}
Oct 30 12:54:07.582235 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:07.581821    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh" event={"ID":"520c958e-79f7-4563-af5d-3f313563da44","Type":"ContainerStarted","Data":"2f9546283bb14118565cf439a0a4fe0c6987c4729635ea776424d2058a6f4346"}
Oct 30 12:54:07.582235 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:07.581904    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"
Oct 30 12:54:07.668055 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:07.668006    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh" podStartSLOduration=1.667992306 podStartE2EDuration="1.667992306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:07.660084924 +0000 UTC m=+489.243514618" watchObservedRunningTime="2026-10-30 12:54:07.667992306 +0000 UTC m=+489.251421999"
Oct 30 12:54:08.529161 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:08.529128    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a13e8d-3947-425a-a5ec-869956621358" path="/var/lib/kubelet/pods/a0a13e8d-3947-425a-a5ec-869956621358/volumes"
Oct 30 12:54:18.587881 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:18.587839    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"
Oct 30 12:54:18.645500 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:54:18.645447    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:54:40.372060 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:40.372023    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z7l6s"]
Oct 30 12:54:40.375500 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:40.375475    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z7l6s"
Oct 30 12:54:40.389244 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:40.389215    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jqdr4\""
Oct 30 12:54:40.400307 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:40.400265    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z7l6s"]
Oct 30 12:54:40.472052 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:40.472014    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76ts8\" (UniqueName: \"kubernetes.io/projected/48b4038f-ec84-4de1-a126-72021d2e2d6b-kube-api-access-76ts8\") pod \"authorino-57f7f8659f-z7l6s\" (UID: \"48b4038f-ec84-4de1-a126-72021d2e2d6b\") " pod="kuadrant-system/authorino-57f7f8659f-z7l6s"
Oct 30 12:54:40.573102 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:40.573069    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76ts8\" (UniqueName: \"kubernetes.io/projected/48b4038f-ec84-4de1-a126-72021d2e2d6b-kube-api-access-76ts8\") pod \"authorino-57f7f8659f-z7l6s\" (UID: \"48b4038f-ec84-4de1-a126-72021d2e2d6b\") " pod="kuadrant-system/authorino-57f7f8659f-z7l6s"
Oct 30 12:54:40.602645 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:40.602616    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76ts8\" (UniqueName: \"kubernetes.io/projected/48b4038f-ec84-4de1-a126-72021d2e2d6b-kube-api-access-76ts8\") pod \"authorino-57f7f8659f-z7l6s\" (UID: \"48b4038f-ec84-4de1-a126-72021d2e2d6b\") " pod="kuadrant-system/authorino-57f7f8659f-z7l6s"
Oct 30 12:54:40.688750 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:40.688671    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z7l6s"
Oct 30 12:54:40.897868 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:40.897837    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qp2rh"]
Oct 30 12:54:40.901572 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:40.901549    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qp2rh"
Oct 30 12:54:40.925129 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:40.925104    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qp2rh"]
Oct 30 12:54:40.977290 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:40.977243    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pgb5\" (UniqueName: \"kubernetes.io/projected/4106fede-a141-4f06-9a07-728096971513-kube-api-access-5pgb5\") pod \"authorino-56c6c9ddd5-qp2rh\" (UID: \"4106fede-a141-4f06-9a07-728096971513\") " pod="kuadrant-system/authorino-56c6c9ddd5-qp2rh"
Oct 30 12:54:40.982390 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:40.982366    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z7l6s"]
Oct 30 12:54:40.987503 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:54:40.987476    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48b4038f_ec84_4de1_a126_72021d2e2d6b.slice/crio-3e99f846fd4ae322f01d64c943d76a89c266d4ede2ea3119550ffcc6d27e6d0b WatchSource:0}: Error finding container 3e99f846fd4ae322f01d64c943d76a89c266d4ede2ea3119550ffcc6d27e6d0b: Status 404 returned error can't find the container with id 3e99f846fd4ae322f01d64c943d76a89c266d4ede2ea3119550ffcc6d27e6d0b
Oct 30 12:54:41.078102 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:41.078066    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pgb5\" (UniqueName: \"kubernetes.io/projected/4106fede-a141-4f06-9a07-728096971513-kube-api-access-5pgb5\") pod \"authorino-56c6c9ddd5-qp2rh\" (UID: \"4106fede-a141-4f06-9a07-728096971513\") " pod="kuadrant-system/authorino-56c6c9ddd5-qp2rh"
Oct 30 12:54:41.099757 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:41.099728    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pgb5\" (UniqueName: \"kubernetes.io/projected/4106fede-a141-4f06-9a07-728096971513-kube-api-access-5pgb5\") pod \"authorino-56c6c9ddd5-qp2rh\" (UID: \"4106fede-a141-4f06-9a07-728096971513\") " pod="kuadrant-system/authorino-56c6c9ddd5-qp2rh"
Oct 30 12:54:41.213794 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:41.213758    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qp2rh"
Oct 30 12:54:41.506815 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:41.506784    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qp2rh"]
Oct 30 12:54:41.512189 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:54:41.512118    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4106fede_a141_4f06_9a07_728096971513.slice/crio-f581831b742b7b639d37ba13e78c6fdadd38fdb9a340ee63d1a7796f5c0e2136 WatchSource:0}: Error finding container f581831b742b7b639d37ba13e78c6fdadd38fdb9a340ee63d1a7796f5c0e2136: Status 404 returned error can't find the container with id f581831b742b7b639d37ba13e78c6fdadd38fdb9a340ee63d1a7796f5c0e2136
Oct 30 12:54:41.730377 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:41.730309    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z7l6s" event={"ID":"48b4038f-ec84-4de1-a126-72021d2e2d6b","Type":"ContainerStarted","Data":"3e99f846fd4ae322f01d64c943d76a89c266d4ede2ea3119550ffcc6d27e6d0b"}
Oct 30 12:54:41.731893 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:41.731863    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qp2rh" event={"ID":"4106fede-a141-4f06-9a07-728096971513","Type":"ContainerStarted","Data":"f581831b742b7b639d37ba13e78c6fdadd38fdb9a340ee63d1a7796f5c0e2136"}
Oct 30 12:54:42.737926 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:42.737875    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z7l6s" event={"ID":"48b4038f-ec84-4de1-a126-72021d2e2d6b","Type":"ContainerStarted","Data":"c8c21cf4815ffab71ece47557d5a08d4f0a3a9402611c19ebe02ca4184ba93f2"}
Oct 30 12:54:42.739438 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:42.739412    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qp2rh" event={"ID":"4106fede-a141-4f06-9a07-728096971513","Type":"ContainerStarted","Data":"8f7059c28df5e74e88fea5e371a3c758425db7e47d2e38861ea848a937f6bd92"}
Oct 30 12:54:42.772996 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:42.772887    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-z7l6s" podStartSLOduration=1.2635575989999999 podStartE2EDuration="2.772872197s" totalImagesPullingTime="1.509314598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:42.772107735 +0000 UTC m=+524.355537442" watchObservedRunningTime="2026-10-30 12:54:42.772872197 +0000 UTC m=+524.356301887"
Oct 30 12:54:42.807577 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:42.807529    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-qp2rh" podStartSLOduration=1.769404662 podStartE2EDuration="2.807513927s" totalImagesPullingTime="1.038109265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:54:42.806566135 +0000 UTC m=+524.389995829" watchObservedRunningTime="2026-10-30 12:54:42.807513927 +0000 UTC m=+524.390943619"
Oct 30 12:54:42.833161 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:42.833132    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z7l6s"]
Oct 30 12:54:44.747472 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:44.747435    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-z7l6s" podUID="48b4038f-ec84-4de1-a126-72021d2e2d6b" containerName="authorino" containerID="cri-o://c8c21cf4815ffab71ece47557d5a08d4f0a3a9402611c19ebe02ca4184ba93f2" gracePeriod=30
Oct 30 12:54:44.996588 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:44.996562    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z7l6s"
Oct 30 12:54:45.115988 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:45.115906    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b4038f-ec84-4de1-a126-72021d2e2d6b-kube-api-access-76ts8\" (UniqueName: \"kubernetes.io/projected/48b4038f-ec84-4de1-a126-72021d2e2d6b-kube-api-access-76ts8\") pod \"48b4038f-ec84-4de1-a126-72021d2e2d6b\" (UID: \"48b4038f-ec84-4de1-a126-72021d2e2d6b\") "
Oct 30 12:54:45.118172 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:45.118147    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b4038f-ec84-4de1-a126-72021d2e2d6b-kube-api-access-76ts8" pod "48b4038f-ec84-4de1-a126-72021d2e2d6b" (UID: "48b4038f-ec84-4de1-a126-72021d2e2d6b"). InnerVolumeSpecName "kube-api-access-76ts8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:54:45.216848 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:45.216810    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76ts8\" (UniqueName: \"kubernetes.io/projected/48b4038f-ec84-4de1-a126-72021d2e2d6b-kube-api-access-76ts8\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:54:45.753095 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:45.753058    5893 generic.go:356] "Generic (PLEG): container finished" podID="48b4038f-ec84-4de1-a126-72021d2e2d6b" containerID="c8c21cf4815ffab71ece47557d5a08d4f0a3a9402611c19ebe02ca4184ba93f2" exitCode=0
Oct 30 12:54:45.753522 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:45.753102    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-z7l6s"
Oct 30 12:54:45.753522 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:45.753152    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z7l6s" event={"ID":"48b4038f-ec84-4de1-a126-72021d2e2d6b","Type":"ContainerDied","Data":"c8c21cf4815ffab71ece47557d5a08d4f0a3a9402611c19ebe02ca4184ba93f2"}
Oct 30 12:54:45.753522 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:45.753190    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-z7l6s" event={"ID":"48b4038f-ec84-4de1-a126-72021d2e2d6b","Type":"ContainerDied","Data":"3e99f846fd4ae322f01d64c943d76a89c266d4ede2ea3119550ffcc6d27e6d0b"}
Oct 30 12:54:45.753522 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:45.753207    5893 scope.go:122] "RemoveContainer" containerID="c8c21cf4815ffab71ece47557d5a08d4f0a3a9402611c19ebe02ca4184ba93f2"
Oct 30 12:54:45.762159 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:45.762139    5893 scope.go:122] "RemoveContainer" containerID="c8c21cf4815ffab71ece47557d5a08d4f0a3a9402611c19ebe02ca4184ba93f2"
Oct 30 12:54:45.762433 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:54:45.762416    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8c21cf4815ffab71ece47557d5a08d4f0a3a9402611c19ebe02ca4184ba93f2\": container with ID starting with c8c21cf4815ffab71ece47557d5a08d4f0a3a9402611c19ebe02ca4184ba93f2 not found: ID does not exist" containerID="c8c21cf4815ffab71ece47557d5a08d4f0a3a9402611c19ebe02ca4184ba93f2"
Oct 30 12:54:45.762477 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:45.762444    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8c21cf4815ffab71ece47557d5a08d4f0a3a9402611c19ebe02ca4184ba93f2"} err="failed to get container status \"c8c21cf4815ffab71ece47557d5a08d4f0a3a9402611c19ebe02ca4184ba93f2\": rpc error: code = NotFound desc = could not find container \"c8c21cf4815ffab71ece47557d5a08d4f0a3a9402611c19ebe02ca4184ba93f2\": container with ID starting with c8c21cf4815ffab71ece47557d5a08d4f0a3a9402611c19ebe02ca4184ba93f2 not found: ID does not exist"
Oct 30 12:54:45.796910 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:45.796881    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z7l6s"]
Oct 30 12:54:45.801066 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:45.801035    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-z7l6s"]
Oct 30 12:54:46.528698 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:54:46.528663    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b4038f-ec84-4de1-a126-72021d2e2d6b" path="/var/lib/kubelet/pods/48b4038f-ec84-4de1-a126-72021d2e2d6b/volumes"
Oct 30 12:54:48.643970 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:54:48.643865    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:55:11.318643 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.318561    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9x292"]
Oct 30 12:55:11.319025 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.318970    5893 cpu_manager.go:414] "containerMap: removing container" podUID="48b4038f-ec84-4de1-a126-72021d2e2d6b" containerName="authorino"
Oct 30 12:55:11.319025 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.318983    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b4038f-ec84-4de1-a126-72021d2e2d6b" containerName="authorino"
Oct 30 12:55:11.319098 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.319066    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b4038f-ec84-4de1-a126-72021d2e2d6b" containerName="authorino"
Oct 30 12:55:11.322032 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.322013    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9x292"
Oct 30 12:55:11.341423 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.341400    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9x292"]
Oct 30 12:55:11.349686 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.349664    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75c4m\" (UniqueName: \"kubernetes.io/projected/6cf5dcb6-df09-4e58-8d85-68f48aff1523-kube-api-access-75c4m\") pod \"authorino-6f6944bcdd-9x292\" (UID: \"6cf5dcb6-df09-4e58-8d85-68f48aff1523\") " pod="kuadrant-system/authorino-6f6944bcdd-9x292"
Oct 30 12:55:11.450230 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.450197    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75c4m\" (UniqueName: \"kubernetes.io/projected/6cf5dcb6-df09-4e58-8d85-68f48aff1523-kube-api-access-75c4m\") pod \"authorino-6f6944bcdd-9x292\" (UID: \"6cf5dcb6-df09-4e58-8d85-68f48aff1523\") " pod="kuadrant-system/authorino-6f6944bcdd-9x292"
Oct 30 12:55:11.472078 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.472042    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75c4m\" (UniqueName: \"kubernetes.io/projected/6cf5dcb6-df09-4e58-8d85-68f48aff1523-kube-api-access-75c4m\") pod \"authorino-6f6944bcdd-9x292\" (UID: \"6cf5dcb6-df09-4e58-8d85-68f48aff1523\") " pod="kuadrant-system/authorino-6f6944bcdd-9x292"
Oct 30 12:55:11.497536 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.497508    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9x292"]
Oct 30 12:55:11.497792 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.497719    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9x292"
Oct 30 12:55:11.528089 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.528060    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cxznd"]
Oct 30 12:55:11.532117 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.532092    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-cxznd"
Oct 30 12:55:11.545063 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.545042    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 12:55:11.553265 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.553237    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cxznd"]
Oct 30 12:55:11.651929 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.651835    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nglbj\" (UniqueName: \"kubernetes.io/projected/1b46becd-44d7-4941-b2a2-69e02096dc09-kube-api-access-nglbj\") pod \"authorino-6889cbd8c4-cxznd\" (UID: \"1b46becd-44d7-4941-b2a2-69e02096dc09\") " pod="kuadrant-system/authorino-6889cbd8c4-cxznd"
Oct 30 12:55:11.651929 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.651889    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1b46becd-44d7-4941-b2a2-69e02096dc09-tls-cert\") pod \"authorino-6889cbd8c4-cxznd\" (UID: \"1b46becd-44d7-4941-b2a2-69e02096dc09\") " pod="kuadrant-system/authorino-6889cbd8c4-cxznd"
Oct 30 12:55:11.685057 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.685024    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cxznd"]
Oct 30 12:55:11.685247 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:55:11.685229    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nglbj tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-cxznd" podUID="1b46becd-44d7-4941-b2a2-69e02096dc09"
Oct 30 12:55:11.712229 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.712199    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c898d64f4-tllg2"]
Oct 30 12:55:11.716317 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.716296    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c898d64f4-tllg2"
Oct 30 12:55:11.735562 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.735531    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c898d64f4-tllg2"]
Oct 30 12:55:11.753479 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.753448    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk5ss\" (UniqueName: \"kubernetes.io/projected/1b563b4e-ff50-4793-a1e8-c4c39f3107c0-kube-api-access-nk5ss\") pod \"authorino-c898d64f4-tllg2\" (UID: \"1b563b4e-ff50-4793-a1e8-c4c39f3107c0\") " pod="kuadrant-system/authorino-c898d64f4-tllg2"
Oct 30 12:55:11.753645 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.753564    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nglbj\" (UniqueName: \"kubernetes.io/projected/1b46becd-44d7-4941-b2a2-69e02096dc09-kube-api-access-nglbj\") pod \"authorino-6889cbd8c4-cxznd\" (UID: \"1b46becd-44d7-4941-b2a2-69e02096dc09\") " pod="kuadrant-system/authorino-6889cbd8c4-cxznd"
Oct 30 12:55:11.753645 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.753599    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1b46becd-44d7-4941-b2a2-69e02096dc09-tls-cert\") pod \"authorino-6889cbd8c4-cxznd\" (UID: \"1b46becd-44d7-4941-b2a2-69e02096dc09\") " pod="kuadrant-system/authorino-6889cbd8c4-cxznd"
Oct 30 12:55:11.753645 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.753627    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1b563b4e-ff50-4793-a1e8-c4c39f3107c0-tls-cert\") pod \"authorino-c898d64f4-tllg2\" (UID: \"1b563b4e-ff50-4793-a1e8-c4c39f3107c0\") " pod="kuadrant-system/authorino-c898d64f4-tllg2"
Oct 30 12:55:11.756362 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.756340    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1b46becd-44d7-4941-b2a2-69e02096dc09-tls-cert\") pod \"authorino-6889cbd8c4-cxznd\" (UID: \"1b46becd-44d7-4941-b2a2-69e02096dc09\") " pod="kuadrant-system/authorino-6889cbd8c4-cxznd"
Oct 30 12:55:11.779804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.779770    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nglbj\" (UniqueName: \"kubernetes.io/projected/1b46becd-44d7-4941-b2a2-69e02096dc09-kube-api-access-nglbj\") pod \"authorino-6889cbd8c4-cxznd\" (UID: \"1b46becd-44d7-4941-b2a2-69e02096dc09\") " pod="kuadrant-system/authorino-6889cbd8c4-cxznd"
Oct 30 12:55:11.797152 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.797126    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9x292"]
Oct 30 12:55:11.802398 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:55:11.802375    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cf5dcb6_df09_4e58_8d85_68f48aff1523.slice/crio-66f793f01957b431f81d7818037aab3dcb4c74ffb7ef7958c2673d8a57191231 WatchSource:0}: Error finding container 66f793f01957b431f81d7818037aab3dcb4c74ffb7ef7958c2673d8a57191231: Status 404 returned error can't find the container with id 66f793f01957b431f81d7818037aab3dcb4c74ffb7ef7958c2673d8a57191231
Oct 30 12:55:11.854088 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.854058    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk5ss\" (UniqueName: \"kubernetes.io/projected/1b563b4e-ff50-4793-a1e8-c4c39f3107c0-kube-api-access-nk5ss\") pod \"authorino-c898d64f4-tllg2\" (UID: \"1b563b4e-ff50-4793-a1e8-c4c39f3107c0\") " pod="kuadrant-system/authorino-c898d64f4-tllg2"
Oct 30 12:55:11.854226 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.854150    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1b563b4e-ff50-4793-a1e8-c4c39f3107c0-tls-cert\") pod \"authorino-c898d64f4-tllg2\" (UID: \"1b563b4e-ff50-4793-a1e8-c4c39f3107c0\") " pod="kuadrant-system/authorino-c898d64f4-tllg2"
Oct 30 12:55:11.856638 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.856618    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1b563b4e-ff50-4793-a1e8-c4c39f3107c0-tls-cert\") pod \"authorino-c898d64f4-tllg2\" (UID: \"1b563b4e-ff50-4793-a1e8-c4c39f3107c0\") " pod="kuadrant-system/authorino-c898d64f4-tllg2"
Oct 30 12:55:11.865265 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.865236    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9x292" event={"ID":"6cf5dcb6-df09-4e58-8d85-68f48aff1523","Type":"ContainerStarted","Data":"66f793f01957b431f81d7818037aab3dcb4c74ffb7ef7958c2673d8a57191231"}
Oct 30 12:55:11.865265 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.865266    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-cxznd"
Oct 30 12:55:11.873593 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.873572    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-cxznd"
Oct 30 12:55:11.875583 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.875566    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk5ss\" (UniqueName: \"kubernetes.io/projected/1b563b4e-ff50-4793-a1e8-c4c39f3107c0-kube-api-access-nk5ss\") pod \"authorino-c898d64f4-tllg2\" (UID: \"1b563b4e-ff50-4793-a1e8-c4c39f3107c0\") " pod="kuadrant-system/authorino-c898d64f4-tllg2"
Oct 30 12:55:11.954584 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.954498    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b46becd-44d7-4941-b2a2-69e02096dc09-kube-api-access-nglbj\" (UniqueName: \"kubernetes.io/projected/1b46becd-44d7-4941-b2a2-69e02096dc09-kube-api-access-nglbj\") pod \"1b46becd-44d7-4941-b2a2-69e02096dc09\" (UID: \"1b46becd-44d7-4941-b2a2-69e02096dc09\") "
Oct 30 12:55:11.954584 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.954537    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b46becd-44d7-4941-b2a2-69e02096dc09-tls-cert\" (UniqueName: \"kubernetes.io/secret/1b46becd-44d7-4941-b2a2-69e02096dc09-tls-cert\") pod \"1b46becd-44d7-4941-b2a2-69e02096dc09\" (UID: \"1b46becd-44d7-4941-b2a2-69e02096dc09\") "
Oct 30 12:55:11.956712 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.956685    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b46becd-44d7-4941-b2a2-69e02096dc09-tls-cert" pod "1b46becd-44d7-4941-b2a2-69e02096dc09" (UID: "1b46becd-44d7-4941-b2a2-69e02096dc09"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:55:11.956827 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:11.956784    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b46becd-44d7-4941-b2a2-69e02096dc09-kube-api-access-nglbj" pod "1b46becd-44d7-4941-b2a2-69e02096dc09" (UID: "1b46becd-44d7-4941-b2a2-69e02096dc09"). InnerVolumeSpecName "kube-api-access-nglbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:12.027946 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:12.027911    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c898d64f4-tllg2"
Oct 30 12:55:12.055900 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:12.055873    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nglbj\" (UniqueName: \"kubernetes.io/projected/1b46becd-44d7-4941-b2a2-69e02096dc09-kube-api-access-nglbj\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:55:12.055900 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:12.055899    5893 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1b46becd-44d7-4941-b2a2-69e02096dc09-tls-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:55:12.322337 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:12.322309    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c898d64f4-tllg2"]
Oct 30 12:55:12.327334 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:55:12.327307    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b563b4e_ff50_4793_a1e8_c4c39f3107c0.slice/crio-134e52ce4627db5f61e60353e215d39c76c2f5dc7e8e633e2ad4c8a9db2b7ff6 WatchSource:0}: Error finding container 134e52ce4627db5f61e60353e215d39c76c2f5dc7e8e633e2ad4c8a9db2b7ff6: Status 404 returned error can't find the container with id 134e52ce4627db5f61e60353e215d39c76c2f5dc7e8e633e2ad4c8a9db2b7ff6
Oct 30 12:55:12.870893 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:12.870803    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c898d64f4-tllg2" event={"ID":"1b563b4e-ff50-4793-a1e8-c4c39f3107c0","Type":"ContainerStarted","Data":"134e52ce4627db5f61e60353e215d39c76c2f5dc7e8e633e2ad4c8a9db2b7ff6"}
Oct 30 12:55:12.872321 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:12.872302    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-cxznd"
Oct 30 12:55:12.872450 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:12.872319    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9x292" event={"ID":"6cf5dcb6-df09-4e58-8d85-68f48aff1523","Type":"ContainerStarted","Data":"eb0f1926d4eecd14d1af349956d5e9723357afcc39cf705a914ab0aed2c9d595"}
Oct 30 12:55:12.872450 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:12.872346    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-9x292" podUID="6cf5dcb6-df09-4e58-8d85-68f48aff1523" containerName="authorino" containerID="cri-o://eb0f1926d4eecd14d1af349956d5e9723357afcc39cf705a914ab0aed2c9d595" gracePeriod=30
Oct 30 12:55:12.907591 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:12.907548    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-9x292" podStartSLOduration=1.477952339 podStartE2EDuration="1.90753416s" totalImagesPullingTime="429.581821ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:12.906644313 +0000 UTC m=+554.490074005" watchObservedRunningTime="2026-10-30 12:55:12.90753416 +0000 UTC m=+554.490963852"
Oct 30 12:55:12.971321 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:12.971297    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cxznd"]
Oct 30 12:55:12.975263 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:12.975241    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-cxznd"]
Oct 30 12:55:13.116728 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:13.116707    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9x292"
Oct 30 12:55:13.163503 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:13.163429    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf5dcb6-df09-4e58-8d85-68f48aff1523-kube-api-access-75c4m\" (UniqueName: \"kubernetes.io/projected/6cf5dcb6-df09-4e58-8d85-68f48aff1523-kube-api-access-75c4m\") pod \"6cf5dcb6-df09-4e58-8d85-68f48aff1523\" (UID: \"6cf5dcb6-df09-4e58-8d85-68f48aff1523\") "
Oct 30 12:55:13.165588 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:13.165559    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf5dcb6-df09-4e58-8d85-68f48aff1523-kube-api-access-75c4m" pod "6cf5dcb6-df09-4e58-8d85-68f48aff1523" (UID: "6cf5dcb6-df09-4e58-8d85-68f48aff1523"). InnerVolumeSpecName "kube-api-access-75c4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:13.264662 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:13.264627    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75c4m\" (UniqueName: \"kubernetes.io/projected/6cf5dcb6-df09-4e58-8d85-68f48aff1523-kube-api-access-75c4m\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:55:13.878312 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:13.878260    5893 generic.go:356] "Generic (PLEG): container finished" podID="6cf5dcb6-df09-4e58-8d85-68f48aff1523" containerID="eb0f1926d4eecd14d1af349956d5e9723357afcc39cf705a914ab0aed2c9d595" exitCode=0
Oct 30 12:55:13.878312 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:13.878298    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9x292" event={"ID":"6cf5dcb6-df09-4e58-8d85-68f48aff1523","Type":"ContainerDied","Data":"eb0f1926d4eecd14d1af349956d5e9723357afcc39cf705a914ab0aed2c9d595"}
Oct 30 12:55:13.878773 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:13.878320    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-9x292"
Oct 30 12:55:13.878773 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:13.878337    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-9x292" event={"ID":"6cf5dcb6-df09-4e58-8d85-68f48aff1523","Type":"ContainerDied","Data":"66f793f01957b431f81d7818037aab3dcb4c74ffb7ef7958c2673d8a57191231"}
Oct 30 12:55:13.878773 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:13.878354    5893 scope.go:122] "RemoveContainer" containerID="eb0f1926d4eecd14d1af349956d5e9723357afcc39cf705a914ab0aed2c9d595"
Oct 30 12:55:13.886950 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:13.886930    5893 scope.go:122] "RemoveContainer" containerID="eb0f1926d4eecd14d1af349956d5e9723357afcc39cf705a914ab0aed2c9d595"
Oct 30 12:55:13.887198 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:55:13.887176    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb0f1926d4eecd14d1af349956d5e9723357afcc39cf705a914ab0aed2c9d595\": container with ID starting with eb0f1926d4eecd14d1af349956d5e9723357afcc39cf705a914ab0aed2c9d595 not found: ID does not exist" containerID="eb0f1926d4eecd14d1af349956d5e9723357afcc39cf705a914ab0aed2c9d595"
Oct 30 12:55:13.887262 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:13.887211    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb0f1926d4eecd14d1af349956d5e9723357afcc39cf705a914ab0aed2c9d595"} err="failed to get container status \"eb0f1926d4eecd14d1af349956d5e9723357afcc39cf705a914ab0aed2c9d595\": rpc error: code = NotFound desc = could not find container \"eb0f1926d4eecd14d1af349956d5e9723357afcc39cf705a914ab0aed2c9d595\": container with ID starting with eb0f1926d4eecd14d1af349956d5e9723357afcc39cf705a914ab0aed2c9d595 not found: ID does not exist"
Oct 30 12:55:13.922881 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:13.922856    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9x292"]
Oct 30 12:55:13.927141 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:13.927116    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-9x292"]
Oct 30 12:55:14.530072 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:14.530037    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b46becd-44d7-4941-b2a2-69e02096dc09" path="/var/lib/kubelet/pods/1b46becd-44d7-4941-b2a2-69e02096dc09/volumes"
Oct 30 12:55:14.530263 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:14.530252    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cf5dcb6-df09-4e58-8d85-68f48aff1523" path="/var/lib/kubelet/pods/6cf5dcb6-df09-4e58-8d85-68f48aff1523/volumes"
Oct 30 12:55:16.893613 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:16.893576    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c898d64f4-tllg2" event={"ID":"1b563b4e-ff50-4793-a1e8-c4c39f3107c0","Type":"ContainerStarted","Data":"9dfaacc058f6b7878be6426f1b9ea320143a2d349787e918ec40f0b4fb85b6d1"}
Oct 30 12:55:16.931728 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:16.931680    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-c898d64f4-tllg2" podStartSLOduration=2.289831239 podStartE2EDuration="5.93166187s" totalImagesPullingTime="3.641830631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:55:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:55:16.930716423 +0000 UTC m=+558.514146116" watchObservedRunningTime="2026-10-30 12:55:16.93166187 +0000 UTC m=+558.515091604"
Oct 30 12:55:16.958067 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:16.958035    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qp2rh"]
Oct 30 12:55:16.958466 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:16.958230    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-qp2rh" podUID="4106fede-a141-4f06-9a07-728096971513" containerName="authorino" containerID="cri-o://8f7059c28df5e74e88fea5e371a3c758425db7e47d2e38861ea848a937f6bd92" gracePeriod=30
Oct 30 12:55:17.209541 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:17.209516    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qp2rh"
Oct 30 12:55:17.297016 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:17.296985    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4106fede-a141-4f06-9a07-728096971513-kube-api-access-5pgb5\" (UniqueName: \"kubernetes.io/projected/4106fede-a141-4f06-9a07-728096971513-kube-api-access-5pgb5\") pod \"4106fede-a141-4f06-9a07-728096971513\" (UID: \"4106fede-a141-4f06-9a07-728096971513\") "
Oct 30 12:55:17.299190 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:17.299163    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4106fede-a141-4f06-9a07-728096971513-kube-api-access-5pgb5" pod "4106fede-a141-4f06-9a07-728096971513" (UID: "4106fede-a141-4f06-9a07-728096971513"). InnerVolumeSpecName "kube-api-access-5pgb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:55:17.398068 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:17.398029    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pgb5\" (UniqueName: \"kubernetes.io/projected/4106fede-a141-4f06-9a07-728096971513-kube-api-access-5pgb5\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:55:17.898694 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:17.898659    5893 generic.go:356] "Generic (PLEG): container finished" podID="4106fede-a141-4f06-9a07-728096971513" containerID="8f7059c28df5e74e88fea5e371a3c758425db7e47d2e38861ea848a937f6bd92" exitCode=0
Oct 30 12:55:17.899157 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:17.898728    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-qp2rh"
Oct 30 12:55:17.899157 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:17.898751    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qp2rh" event={"ID":"4106fede-a141-4f06-9a07-728096971513","Type":"ContainerDied","Data":"8f7059c28df5e74e88fea5e371a3c758425db7e47d2e38861ea848a937f6bd92"}
Oct 30 12:55:17.899157 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:17.898796    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-qp2rh" event={"ID":"4106fede-a141-4f06-9a07-728096971513","Type":"ContainerDied","Data":"f581831b742b7b639d37ba13e78c6fdadd38fdb9a340ee63d1a7796f5c0e2136"}
Oct 30 12:55:17.899157 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:17.898812    5893 scope.go:122] "RemoveContainer" containerID="8f7059c28df5e74e88fea5e371a3c758425db7e47d2e38861ea848a937f6bd92"
Oct 30 12:55:17.907676 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:17.907659    5893 scope.go:122] "RemoveContainer" containerID="8f7059c28df5e74e88fea5e371a3c758425db7e47d2e38861ea848a937f6bd92"
Oct 30 12:55:17.907971 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:55:17.907950    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7059c28df5e74e88fea5e371a3c758425db7e47d2e38861ea848a937f6bd92\": container with ID starting with 8f7059c28df5e74e88fea5e371a3c758425db7e47d2e38861ea848a937f6bd92 not found: ID does not exist" containerID="8f7059c28df5e74e88fea5e371a3c758425db7e47d2e38861ea848a937f6bd92"
Oct 30 12:55:17.908039 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:17.907984    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7059c28df5e74e88fea5e371a3c758425db7e47d2e38861ea848a937f6bd92"} err="failed to get container status \"8f7059c28df5e74e88fea5e371a3c758425db7e47d2e38861ea848a937f6bd92\": rpc error: code = NotFound desc = could not find container \"8f7059c28df5e74e88fea5e371a3c758425db7e47d2e38861ea848a937f6bd92\": container with ID starting with 8f7059c28df5e74e88fea5e371a3c758425db7e47d2e38861ea848a937f6bd92 not found: ID does not exist"
Oct 30 12:55:17.944895 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:17.944868    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qp2rh"]
Oct 30 12:55:17.955098 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:17.955069    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-qp2rh"]
Oct 30 12:55:18.529071 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:18.529043    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4106fede-a141-4f06-9a07-728096971513" path="/var/lib/kubelet/pods/4106fede-a141-4f06-9a07-728096971513/volumes"
Oct 30 12:55:18.643958 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:55:18.643925    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:55:48.644661 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:55:48.644623    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:55:58.574105 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:58.574079    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 12:55:58.575082 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:58.575066    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 12:55:58.586299 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:58.586257    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 12:55:58.587862 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:55:58.587842    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 12:56:01.108739 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.108707    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"]
Oct 30 12:56:01.109122 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.109051    5893 cpu_manager.go:414] "containerMap: removing container" podUID="6cf5dcb6-df09-4e58-8d85-68f48aff1523" containerName="authorino"
Oct 30 12:56:01.109122 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.109061    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf5dcb6-df09-4e58-8d85-68f48aff1523" containerName="authorino"
Oct 30 12:56:01.109122 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.109077    5893 cpu_manager.go:414] "containerMap: removing container" podUID="4106fede-a141-4f06-9a07-728096971513" containerName="authorino"
Oct 30 12:56:01.109122 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.109083    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4106fede-a141-4f06-9a07-728096971513" containerName="authorino"
Oct 30 12:56:01.109370 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.109237    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf5dcb6-df09-4e58-8d85-68f48aff1523" containerName="authorino"
Oct 30 12:56:01.109370 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.109250    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="4106fede-a141-4f06-9a07-728096971513" containerName="authorino"
Oct 30 12:56:01.113844 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.113825    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:01.143220 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.143191    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nb7kb\""
Oct 30 12:56:01.143220 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.143218    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 12:56:01.143497 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.143192    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 12:56:01.143497 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.143294    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 12:56:01.143497 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.143301    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 12:56:01.144342 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.144318    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"]
Oct 30 12:56:01.178486 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.178454    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4380a330-1cdb-42b1-ab66-50f9b6595b84-openshift-service-ca\") pod \"maas-api-5c48877c75-lr5lp\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:01.178486 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.178492    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f948\" (UniqueName: \"kubernetes.io/projected/4380a330-1cdb-42b1-ab66-50f9b6595b84-kube-api-access-5f948\") pod \"maas-api-5c48877c75-lr5lp\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:01.178718 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.178589    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4380a330-1cdb-42b1-ab66-50f9b6595b84-maas-api-tls\") pod \"maas-api-5c48877c75-lr5lp\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:01.178718 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.178650    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4380a330-1cdb-42b1-ab66-50f9b6595b84-metrics-certs\") pod \"maas-api-5c48877c75-lr5lp\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:01.279722 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.279682    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4380a330-1cdb-42b1-ab66-50f9b6595b84-openshift-service-ca\") pod \"maas-api-5c48877c75-lr5lp\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:01.279722 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.279724    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f948\" (UniqueName: \"kubernetes.io/projected/4380a330-1cdb-42b1-ab66-50f9b6595b84-kube-api-access-5f948\") pod \"maas-api-5c48877c75-lr5lp\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:01.279997 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.279796    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4380a330-1cdb-42b1-ab66-50f9b6595b84-maas-api-tls\") pod \"maas-api-5c48877c75-lr5lp\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:01.279997 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.279823    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4380a330-1cdb-42b1-ab66-50f9b6595b84-metrics-certs\") pod \"maas-api-5c48877c75-lr5lp\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:01.280348 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.280324    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4380a330-1cdb-42b1-ab66-50f9b6595b84-openshift-service-ca\") pod \"maas-api-5c48877c75-lr5lp\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:01.282257 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.282235    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4380a330-1cdb-42b1-ab66-50f9b6595b84-maas-api-tls\") pod \"maas-api-5c48877c75-lr5lp\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:01.282451 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.282430    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4380a330-1cdb-42b1-ab66-50f9b6595b84-metrics-certs\") pod \"maas-api-5c48877c75-lr5lp\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:01.300921 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.300890    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f948\" (UniqueName: \"kubernetes.io/projected/4380a330-1cdb-42b1-ab66-50f9b6595b84-kube-api-access-5f948\") pod \"maas-api-5c48877c75-lr5lp\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:01.426707 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.426634    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:01.722235 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:01.722199    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"]
Oct 30 12:56:01.727845 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:56:01.727815    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4380a330_1cdb_42b1_ab66_50f9b6595b84.slice/crio-912386c571855d07f460801424b6ea9d2ac2ba83bf3b6abeaceb878e35a6fbd7 WatchSource:0}: Error finding container 912386c571855d07f460801424b6ea9d2ac2ba83bf3b6abeaceb878e35a6fbd7: Status 404 returned error can't find the container with id 912386c571855d07f460801424b6ea9d2ac2ba83bf3b6abeaceb878e35a6fbd7
Oct 30 12:56:02.095064 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:02.094980    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp" event={"ID":"4380a330-1cdb-42b1-ab66-50f9b6595b84","Type":"ContainerStarted","Data":"912386c571855d07f460801424b6ea9d2ac2ba83bf3b6abeaceb878e35a6fbd7"}
Oct 30 12:56:05.111457 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:05.111409    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp" event={"ID":"4380a330-1cdb-42b1-ab66-50f9b6595b84","Type":"ContainerStarted","Data":"a4f91eef0cd582c4b4dea864971e42860808a87abebf6ecce064b3de907075a2"}
Oct 30 12:56:05.111997 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:05.111564    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:05.151387 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:05.151335    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp" podStartSLOduration=1.704805151 podStartE2EDuration="4.151321763s" totalImagesPullingTime="2.446516612s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:05.150941277 +0000 UTC m=+606.734370970" watchObservedRunningTime="2026-10-30 12:56:05.151321763 +0000 UTC m=+606.734751455"
Oct 30 12:56:08.974642 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:08.974613    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"]
Oct 30 12:56:08.978416 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:08.978391    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:08.995994 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:08.995958    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 12:56:08.995994 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:08.995972    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4s79t\""
Oct 30 12:56:09.007183 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.007158    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"]
Oct 30 12:56:09.054351 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.054315    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.054351 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.054351    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.054583 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.054379    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.054583 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.054475    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.054583 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.054490    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.054583 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.054518    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.054583 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.054541    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpgts\" (UniqueName: \"kubernetes.io/projected/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-kube-api-access-lpgts\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.054790 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.054672    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.054790 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.054699    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.155851 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.155813    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpgts\" (UniqueName: \"kubernetes.io/projected/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-kube-api-access-lpgts\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.155851 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.155855    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.156124 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.155873    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.156124 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.155918    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.156124 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.155935    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.156124 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.155962    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.156367 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.156141    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.156367 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.156183    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.156367 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.156236    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.156530 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.156369    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.156530 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.156388    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.156631 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.156594    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.156631 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.156621    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.156927 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.156901    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.158426 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.158405    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.158719 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.158700    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.177504 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.177478    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpgts\" (UniqueName: \"kubernetes.io/projected/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-kube-api-access-lpgts\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.177630 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.177482    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-8982g\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.291041 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.290944    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:09.606020 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:09.605929    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"]
Oct 30 12:56:09.610946 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:56:09.610917    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14fbc94f_88db_47d5_9cfe_d59e8e0c9874.slice/crio-365e18e8a519f9438c8acb9bf9b1aefdddc594a362f1b1534812fc4bb8d521f9 WatchSource:0}: Error finding container 365e18e8a519f9438c8acb9bf9b1aefdddc594a362f1b1534812fc4bb8d521f9: Status 404 returned error can't find the container with id 365e18e8a519f9438c8acb9bf9b1aefdddc594a362f1b1534812fc4bb8d521f9
Oct 30 12:56:10.133651 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:10.133615    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g" event={"ID":"14fbc94f-88db-47d5-9cfe-d59e8e0c9874","Type":"ContainerStarted","Data":"365e18e8a519f9438c8acb9bf9b1aefdddc594a362f1b1534812fc4bb8d521f9"}
Oct 30 12:56:11.121932 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:11.121901    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 12:56:12.350126 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:12.350096    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:12.350408 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:12.350164    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:12.350408 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:12.350194    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:56:13.169326 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:13.169296    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g" event={"ID":"14fbc94f-88db-47d5-9cfe-d59e8e0c9874","Type":"ContainerStarted","Data":"cff9e3e26710e9b3415ecfc8acda2ffa8d6fa8a18cdadadfa2aaa2b8fc962f59"}
Oct 30 12:56:13.221968 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:13.221917    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g" podStartSLOduration=2.4854769279999998 podStartE2EDuration="5.22190317s" totalImagesPullingTime="2.736426242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:56:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:56:13.221089099 +0000 UTC m=+614.804518786" watchObservedRunningTime="2026-10-30 12:56:13.22190317 +0000 UTC m=+614.805332862"
Oct 30 12:56:13.291331 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:13.291295    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:13.292808 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:13.292782    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8982g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:56:13.292937 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:13.292829    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g" podUID="14fbc94f-88db-47d5-9cfe-d59e8e0c9874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 12:56:14.291979 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:14.291938    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8982g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:56:14.292412 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:14.292002    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g" podUID="14fbc94f-88db-47d5-9cfe-d59e8e0c9874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 12:56:15.291731 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:15.291694    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8982g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:56:15.292064 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:15.291755    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g" podUID="14fbc94f-88db-47d5-9cfe-d59e8e0c9874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 12:56:16.292098 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:16.292060    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8982g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:56:16.292539 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:16.292119    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g" podUID="14fbc94f-88db-47d5-9cfe-d59e8e0c9874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 12:56:17.291979 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:17.291942    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8982g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:56:17.292160 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:17.292001    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g" podUID="14fbc94f-88db-47d5-9cfe-d59e8e0c9874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 12:56:18.291545 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:18.291511    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8982g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Oct 30 12:56:18.291747 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:18.291577    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g" podUID="14fbc94f-88db-47d5-9cfe-d59e8e0c9874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Oct 30 12:56:18.645022 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:56:18.644936    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:56:19.295944 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:19.295903    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:19.296196 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:19.296176    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:19.296780 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:56:19.296760    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 12:56:48.645210 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:56:48.645120    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:57:09.809719 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:09.809675    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"]
Oct 30 12:57:09.814107 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:09.814085    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:09.834666 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:09.834640    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 12:57:09.834918 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:09.834906    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 12:57:09.840974 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:09.840947    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-58g6n\""
Oct 30 12:57:09.841141 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:09.841099    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 12:57:09.846233 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:09.846211    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"]
Oct 30 12:57:09.932865 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:09.932812    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/387d629f-47f2-40b9-85f5-ff8d42310c88-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:09.932865 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:09.932847    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/387d629f-47f2-40b9-85f5-ff8d42310c88-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:09.933110 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:09.932872    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpz4z\" (UniqueName: \"kubernetes.io/projected/387d629f-47f2-40b9-85f5-ff8d42310c88-kube-api-access-fpz4z\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:09.933110 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:09.933059    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/387d629f-47f2-40b9-85f5-ff8d42310c88-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:09.933110 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:09.933090    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387d629f-47f2-40b9-85f5-ff8d42310c88-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:09.933243 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:09.933169    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/387d629f-47f2-40b9-85f5-ff8d42310c88-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:10.034320 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.034266    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/387d629f-47f2-40b9-85f5-ff8d42310c88-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:10.034507 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.034334    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387d629f-47f2-40b9-85f5-ff8d42310c88-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:10.034507 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.034358    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/387d629f-47f2-40b9-85f5-ff8d42310c88-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:10.034507 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.034399    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/387d629f-47f2-40b9-85f5-ff8d42310c88-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:10.034678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.034526    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/387d629f-47f2-40b9-85f5-ff8d42310c88-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:10.034678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.034565    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpz4z\" (UniqueName: \"kubernetes.io/projected/387d629f-47f2-40b9-85f5-ff8d42310c88-kube-api-access-fpz4z\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:10.034798 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.034772    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/387d629f-47f2-40b9-85f5-ff8d42310c88-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:10.034798 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.034790    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/387d629f-47f2-40b9-85f5-ff8d42310c88-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:10.034926 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.034905    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/387d629f-47f2-40b9-85f5-ff8d42310c88-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:10.037001 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.036972    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/387d629f-47f2-40b9-85f5-ff8d42310c88-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:10.037463 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.037446    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/387d629f-47f2-40b9-85f5-ff8d42310c88-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:10.057311 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.057253    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpz4z\" (UniqueName: \"kubernetes.io/projected/387d629f-47f2-40b9-85f5-ff8d42310c88-kube-api-access-fpz4z\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-2cmc8\" (UID: \"387d629f-47f2-40b9-85f5-ff8d42310c88\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:10.133159 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.133056    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:10.457074 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.456973    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"]
Oct 30 12:57:10.470445 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:10.470421    5893 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 12:57:11.432298 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.432241    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8" event={"ID":"387d629f-47f2-40b9-85f5-ff8d42310c88","Type":"ContainerStarted","Data":"6f88d61cdc710b7fa28e533e6f6d3bb65708b7bf8a9bae12bf8516990b170af6"}
Oct 30 12:57:11.623210 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.623181    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"]
Oct 30 12:57:11.626690 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.626670    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.651942 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.651886    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 30 12:57:11.653306 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.653265    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"]
Oct 30 12:57:11.755520 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.755493    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.755682 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.755530    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.755682 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.755601    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.755797 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.755766    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdtn7\" (UniqueName: \"kubernetes.io/projected/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-kube-api-access-jdtn7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.755853 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.755795    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.755853 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.755838    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.856693 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.856663    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.856890 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.856703    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.856890 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.856736    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.856890 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.856832    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdtn7\" (UniqueName: \"kubernetes.io/projected/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-kube-api-access-jdtn7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.856890 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.856860    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.857117 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.856895    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.857298 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.857236    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.857413 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.857326    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.857413 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.857375    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.859142 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.859126    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.859425 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.859408    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.878540 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.878513    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdtn7\" (UniqueName: \"kubernetes.io/projected/d9296ecf-4e97-4ddb-9ef4-9d136b7317c9-kube-api-access-jdtn7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f94xm\" (UID: \"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:11.941200 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:11.941109    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:12.243019 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:12.242982    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"]
Oct 30 12:57:12.247342 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:57:12.247315    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9296ecf_4e97_4ddb_9ef4_9d136b7317c9.slice/crio-ca5c0a60f2d51ce183a111473bb88b80feac8d756932eba8c2d9e182370b0844 WatchSource:0}: Error finding container ca5c0a60f2d51ce183a111473bb88b80feac8d756932eba8c2d9e182370b0844: Status 404 returned error can't find the container with id ca5c0a60f2d51ce183a111473bb88b80feac8d756932eba8c2d9e182370b0844
Oct 30 12:57:12.439173 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:12.439133    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8" event={"ID":"387d629f-47f2-40b9-85f5-ff8d42310c88","Type":"ContainerStarted","Data":"51c546a9d6b4a329ba620402730a03bab7a5148f0a3b98b6125fd80d5ab906f7"}
Oct 30 12:57:12.440476 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:12.440440    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm" event={"ID":"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9","Type":"ContainerStarted","Data":"ca5c0a60f2d51ce183a111473bb88b80feac8d756932eba8c2d9e182370b0844"}
Oct 30 12:57:12.484469 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:12.484413    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8" podStartSLOduration=2.285753779 podStartE2EDuration="3.484393684s" totalImagesPullingTime="1.198639905s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:12.483571332 +0000 UTC m=+674.067001040" watchObservedRunningTime="2026-10-30 12:57:12.484393684 +0000 UTC m=+674.067823377"
Oct 30 12:57:13.135727 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:13.135390    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:13.450838 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:13.450747    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm" event={"ID":"d9296ecf-4e97-4ddb-9ef4-9d136b7317c9","Type":"ContainerStarted","Data":"084073b1d12e1ca8c2f6ca5f8ac824a4df10360333193c85de2c3e1de5245b8a"}
Oct 30 12:57:13.503768 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:13.503707    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm" podStartSLOduration=2.289552606 podStartE2EDuration="2.5036893s" totalImagesPullingTime="214.136694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:13.503585445 +0000 UTC m=+675.087015162" watchObservedRunningTime="2026-10-30 12:57:13.5036893 +0000 UTC m=+675.087118993"
Oct 30 12:57:13.942298 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:13.941384    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:17.438456 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.436569    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"]
Oct 30 12:57:17.445056 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.445019    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.474957 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.474933    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 30 12:57:17.476863 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.476753    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"]
Oct 30 12:57:17.618812 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.618770    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66ed2849-eb53-426f-b0c5-5e059453882a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.619007 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.618905    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66ed2849-eb53-426f-b0c5-5e059453882a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.619007 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.618948    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vnjd\" (UniqueName: \"kubernetes.io/projected/66ed2849-eb53-426f-b0c5-5e059453882a-kube-api-access-7vnjd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.619007 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.618978    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ed2849-eb53-426f-b0c5-5e059453882a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.619145 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.619104    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66ed2849-eb53-426f-b0c5-5e059453882a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.619197 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.619144    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66ed2849-eb53-426f-b0c5-5e059453882a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.719933 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.719874    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66ed2849-eb53-426f-b0c5-5e059453882a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.720128 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.719942    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vnjd\" (UniqueName: \"kubernetes.io/projected/66ed2849-eb53-426f-b0c5-5e059453882a-kube-api-access-7vnjd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.720128 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.719973    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ed2849-eb53-426f-b0c5-5e059453882a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.720128 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.720050    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66ed2849-eb53-426f-b0c5-5e059453882a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.720128 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.720069    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66ed2849-eb53-426f-b0c5-5e059453882a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.720411 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.720153    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66ed2849-eb53-426f-b0c5-5e059453882a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.721191 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.720802    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66ed2849-eb53-426f-b0c5-5e059453882a-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.721636 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.721614    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66ed2849-eb53-426f-b0c5-5e059453882a-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.722447 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.722422    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66ed2849-eb53-426f-b0c5-5e059453882a-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.723116 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.723077    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66ed2849-eb53-426f-b0c5-5e059453882a-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.724410 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.724384    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66ed2849-eb53-426f-b0c5-5e059453882a-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.741873 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.741844    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vnjd\" (UniqueName: \"kubernetes.io/projected/66ed2849-eb53-426f-b0c5-5e059453882a-kube-api-access-7vnjd\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65\" (UID: \"66ed2849-eb53-426f-b0c5-5e059453882a\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:17.763507 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:17.763477    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:18.076545 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:18.076459    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"]
Oct 30 12:57:18.082239 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:57:18.082208    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ed2849_eb53_426f_b0c5_5e059453882a.slice/crio-cce5c5362001f2bff1188cf60b4e0e543a9ad6cf4dbc6fcfb0bf196f6c60a5ee WatchSource:0}: Error finding container cce5c5362001f2bff1188cf60b4e0e543a9ad6cf4dbc6fcfb0bf196f6c60a5ee: Status 404 returned error can't find the container with id cce5c5362001f2bff1188cf60b4e0e543a9ad6cf4dbc6fcfb0bf196f6c60a5ee
Oct 30 12:57:18.483804 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:18.483757    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65" event={"ID":"66ed2849-eb53-426f-b0c5-5e059453882a","Type":"ContainerStarted","Data":"d1804ed0aaa2480c80328c5c46f190fde2a7d3d0c15bc5b86fb8676f1283fb9b"}
Oct 30 12:57:18.484340 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:18.483822    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65" event={"ID":"66ed2849-eb53-426f-b0c5-5e059453882a","Type":"ContainerStarted","Data":"cce5c5362001f2bff1188cf60b4e0e543a9ad6cf4dbc6fcfb0bf196f6c60a5ee"}
Oct 30 12:57:18.524045 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:18.523997    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65" podStartSLOduration=1.308345589 podStartE2EDuration="1.523978815s" totalImagesPullingTime="215.633226ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:18.522626201 +0000 UTC m=+680.106055912" watchObservedRunningTime="2026-10-30 12:57:18.523978815 +0000 UTC m=+680.107408508"
Oct 30 12:57:18.646467 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:57:18.646379    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:57:18.764288 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:18.764242    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:20.133665 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.133632    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:20.147109 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.147084    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:20.340137 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.340102    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"]
Oct 30 12:57:20.346137 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.346114    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.370494 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.370446    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 12:57:20.373229 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.373204    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"]
Oct 30 12:57:20.445237 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.445138    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8dmh\" (UniqueName: \"kubernetes.io/projected/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-kube-api-access-n8dmh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.445237 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.445197    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.445513 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.445282    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.445513 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.445309    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.445513 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.445342    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.445513 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.445361    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.511175 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.511143    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-2cmc8"
Oct 30 12:57:20.546620 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.546577    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.546819 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.546678    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.546819 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.546707    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.546819 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.546776    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.547000 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.546824    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.547000 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.546924    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8dmh\" (UniqueName: \"kubernetes.io/projected/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-kube-api-access-n8dmh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.547117 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.547005    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.547313 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.547256    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.547880 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.547856    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.550078 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.550056    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.553049 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.553026    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.592814 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.592775    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8dmh\" (UniqueName: \"kubernetes.io/projected/e2b6f77b-f0bc-4e73-9572-2e652b9f58ba-kube-api-access-n8dmh\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd\" (UID: \"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.659614 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.659302    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:20.981103 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:20.981012    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"]
Oct 30 12:57:20.992961 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:57:20.992881    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2b6f77b_f0bc_4e73_9572_2e652b9f58ba.slice/crio-f8c617c5521496e37193b28285174bc8a6e7472331112fb4566186c6e9bc9663 WatchSource:0}: Error finding container f8c617c5521496e37193b28285174bc8a6e7472331112fb4566186c6e9bc9663: Status 404 returned error can't find the container with id f8c617c5521496e37193b28285174bc8a6e7472331112fb4566186c6e9bc9663
Oct 30 12:57:21.509599 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:21.509513    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" event={"ID":"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba","Type":"ContainerStarted","Data":"c0f35546668405dfb7d6616b02474a23e951e4fed0f762fe39d61df93c5a92c0"}
Oct 30 12:57:21.509599 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:21.509564    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" event={"ID":"e2b6f77b-f0bc-4e73-9572-2e652b9f58ba","Type":"ContainerStarted","Data":"f8c617c5521496e37193b28285174bc8a6e7472331112fb4566186c6e9bc9663"}
Oct 30 12:57:21.549844 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:21.549782    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podStartSLOduration=1.312631887 podStartE2EDuration="1.549762943s" totalImagesPullingTime="237.131056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:21.548261251 +0000 UTC m=+683.131690943" watchObservedRunningTime="2026-10-30 12:57:21.549762943 +0000 UTC m=+683.133192636"
Oct 30 12:57:21.660145 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:21.660108    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:21.941832 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:21.941750    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:21.954014 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:21.953990    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:22.530608 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:22.530582    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f94xm"
Oct 30 12:57:27.763681 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:27.763646    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:27.776294 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:27.776255    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:28.567311 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:28.567265    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65"
Oct 30 12:57:30.660435 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:30.660404    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:30.672764 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:30.672737    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:31.581497 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:31.581464    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 12:57:37.812743 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:37.812709    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-845dc99f6f-cc7nr"]
Oct 30 12:57:37.819065 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:37.819045    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-845dc99f6f-cc7nr"
Oct 30 12:57:37.838460 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:37.838433    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-845dc99f6f-cc7nr"]
Oct 30 12:57:37.939608 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:37.939572    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1a839f3-745c-4270-a3ff-0145e0cd0ebc-tls-cert\") pod \"authorino-845dc99f6f-cc7nr\" (UID: \"f1a839f3-745c-4270-a3ff-0145e0cd0ebc\") " pod="kuadrant-system/authorino-845dc99f6f-cc7nr"
Oct 30 12:57:37.939785 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:37.939688    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7ptf\" (UniqueName: \"kubernetes.io/projected/f1a839f3-745c-4270-a3ff-0145e0cd0ebc-kube-api-access-n7ptf\") pod \"authorino-845dc99f6f-cc7nr\" (UID: \"f1a839f3-745c-4270-a3ff-0145e0cd0ebc\") " pod="kuadrant-system/authorino-845dc99f6f-cc7nr"
Oct 30 12:57:38.040205 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:38.040169    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1a839f3-745c-4270-a3ff-0145e0cd0ebc-tls-cert\") pod \"authorino-845dc99f6f-cc7nr\" (UID: \"f1a839f3-745c-4270-a3ff-0145e0cd0ebc\") " pod="kuadrant-system/authorino-845dc99f6f-cc7nr"
Oct 30 12:57:38.040429 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:38.040304    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7ptf\" (UniqueName: \"kubernetes.io/projected/f1a839f3-745c-4270-a3ff-0145e0cd0ebc-kube-api-access-n7ptf\") pod \"authorino-845dc99f6f-cc7nr\" (UID: \"f1a839f3-745c-4270-a3ff-0145e0cd0ebc\") " pod="kuadrant-system/authorino-845dc99f6f-cc7nr"
Oct 30 12:57:38.043081 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:38.043054    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f1a839f3-745c-4270-a3ff-0145e0cd0ebc-tls-cert\") pod \"authorino-845dc99f6f-cc7nr\" (UID: \"f1a839f3-745c-4270-a3ff-0145e0cd0ebc\") " pod="kuadrant-system/authorino-845dc99f6f-cc7nr"
Oct 30 12:57:38.062580 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:38.062551    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7ptf\" (UniqueName: \"kubernetes.io/projected/f1a839f3-745c-4270-a3ff-0145e0cd0ebc-kube-api-access-n7ptf\") pod \"authorino-845dc99f6f-cc7nr\" (UID: \"f1a839f3-745c-4270-a3ff-0145e0cd0ebc\") " pod="kuadrant-system/authorino-845dc99f6f-cc7nr"
Oct 30 12:57:38.132420 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:38.132346    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-845dc99f6f-cc7nr"
Oct 30 12:57:38.433063 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:38.432980    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-845dc99f6f-cc7nr"]
Oct 30 12:57:38.438159 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:57:38.438136    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1a839f3_745c_4270_a3ff_0145e0cd0ebc.slice/crio-ef3bc154611c2904aaa079d081f688ebe97a6cec34ef58f82560c0a8fa37314a WatchSource:0}: Error finding container ef3bc154611c2904aaa079d081f688ebe97a6cec34ef58f82560c0a8fa37314a: Status 404 returned error can't find the container with id ef3bc154611c2904aaa079d081f688ebe97a6cec34ef58f82560c0a8fa37314a
Oct 30 12:57:38.603957 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:38.603925    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-845dc99f6f-cc7nr" event={"ID":"f1a839f3-745c-4270-a3ff-0145e0cd0ebc","Type":"ContainerStarted","Data":"ef3bc154611c2904aaa079d081f688ebe97a6cec34ef58f82560c0a8fa37314a"}
Oct 30 12:57:39.269244 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.269211    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"]
Oct 30 12:57:39.273167 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.273144    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.302774 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.302610    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"]
Oct 30 12:57:39.352850 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.352764    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.352850 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.352811    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k627m\" (UniqueName: \"kubernetes.io/projected/44ac65bd-b10a-477b-b196-9ffe6a902ca0-kube-api-access-k627m\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.353084 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.352902    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.353084 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.352941    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.353084 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.352982    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.353084 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.353045    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.353241 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.353181    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.353241 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.353216    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.353375 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.353257    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.454796 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.454759    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.454938 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.454806    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k627m\" (UniqueName: \"kubernetes.io/projected/44ac65bd-b10a-477b-b196-9ffe6a902ca0-kube-api-access-k627m\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.454938 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.454848    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.454938 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.454886    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.455068 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.455034    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.455121 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.455093    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.455176 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.455160    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.455227 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.455213    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.455311 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.455293    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.455424 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.455397    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.455600 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.455473    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.455689 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.455532    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.455901 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.455874    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.455972 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.455875    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.457702 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.457662    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.457814 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.457795    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.479829 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.479793    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k627m\" (UniqueName: \"kubernetes.io/projected/44ac65bd-b10a-477b-b196-9ffe6a902ca0-kube-api-access-k627m\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.479998 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.479803    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-qg8wr\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.585973 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.585941    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:39.611082 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.611000    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-845dc99f6f-cc7nr" event={"ID":"f1a839f3-745c-4270-a3ff-0145e0cd0ebc","Type":"ContainerStarted","Data":"c865a197dae3dbd9b3a1208e0336a06ff74eb701094dc51848bfb14cfe7cec44"}
Oct 30 12:57:39.705032 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.704976    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-845dc99f6f-cc7nr" podStartSLOduration=2.047914629 podStartE2EDuration="2.70495979s" totalImagesPullingTime="657.045161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:39.657230625 +0000 UTC m=+701.240660318" watchObservedRunningTime="2026-10-30 12:57:39.70495979 +0000 UTC m=+701.288389489"
Oct 30 12:57:39.706418 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.706393    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c898d64f4-tllg2"]
Oct 30 12:57:39.707370 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.706659    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-c898d64f4-tllg2" podUID="1b563b4e-ff50-4793-a1e8-c4c39f3107c0" containerName="authorino" containerID="cri-o://9dfaacc058f6b7878be6426f1b9ea320143a2d349787e918ec40f0b4fb85b6d1" gracePeriod=30
Oct 30 12:57:39.933662 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.933626    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"]
Oct 30 12:57:39.938590 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:57:39.938551    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ac65bd_b10a_477b_b196_9ffe6a902ca0.slice/crio-20e2bf9cbb3622f50c04028b150a9b92a886fcec86fc324cf24be9bbe1097163 WatchSource:0}: Error finding container 20e2bf9cbb3622f50c04028b150a9b92a886fcec86fc324cf24be9bbe1097163: Status 404 returned error can't find the container with id 20e2bf9cbb3622f50c04028b150a9b92a886fcec86fc324cf24be9bbe1097163
Oct 30 12:57:39.941224 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.941187    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:39.941322 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.941294    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:39.941374 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:39.941342    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:40.001890 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.001868    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c898d64f4-tllg2"
Oct 30 12:57:40.162743 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.162645    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b563b4e-ff50-4793-a1e8-c4c39f3107c0-tls-cert\" (UniqueName: \"kubernetes.io/secret/1b563b4e-ff50-4793-a1e8-c4c39f3107c0-tls-cert\") pod \"1b563b4e-ff50-4793-a1e8-c4c39f3107c0\" (UID: \"1b563b4e-ff50-4793-a1e8-c4c39f3107c0\") "
Oct 30 12:57:40.162743 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.162701    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b563b4e-ff50-4793-a1e8-c4c39f3107c0-kube-api-access-nk5ss\" (UniqueName: \"kubernetes.io/projected/1b563b4e-ff50-4793-a1e8-c4c39f3107c0-kube-api-access-nk5ss\") pod \"1b563b4e-ff50-4793-a1e8-c4c39f3107c0\" (UID: \"1b563b4e-ff50-4793-a1e8-c4c39f3107c0\") "
Oct 30 12:57:40.164931 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.164904    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b563b4e-ff50-4793-a1e8-c4c39f3107c0-kube-api-access-nk5ss" pod "1b563b4e-ff50-4793-a1e8-c4c39f3107c0" (UID: "1b563b4e-ff50-4793-a1e8-c4c39f3107c0"). InnerVolumeSpecName "kube-api-access-nk5ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 12:57:40.173666 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.173642    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b563b4e-ff50-4793-a1e8-c4c39f3107c0-tls-cert" pod "1b563b4e-ff50-4793-a1e8-c4c39f3107c0" (UID: "1b563b4e-ff50-4793-a1e8-c4c39f3107c0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 12:57:40.264181 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.264144    5893 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1b563b4e-ff50-4793-a1e8-c4c39f3107c0-tls-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:57:40.264181 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.264174    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk5ss\" (UniqueName: \"kubernetes.io/projected/1b563b4e-ff50-4793-a1e8-c4c39f3107c0-kube-api-access-nk5ss\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 12:57:40.617730 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.617693    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" event={"ID":"44ac65bd-b10a-477b-b196-9ffe6a902ca0","Type":"ContainerStarted","Data":"fe7413e697a8f873b64c41e418c86ee426a4950be709f1aba33aeeb10952325a"}
Oct 30 12:57:40.618210 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.617758    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" event={"ID":"44ac65bd-b10a-477b-b196-9ffe6a902ca0","Type":"ContainerStarted","Data":"20e2bf9cbb3622f50c04028b150a9b92a886fcec86fc324cf24be9bbe1097163"}
Oct 30 12:57:40.619183 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.619154    5893 generic.go:356] "Generic (PLEG): container finished" podID="1b563b4e-ff50-4793-a1e8-c4c39f3107c0" containerID="9dfaacc058f6b7878be6426f1b9ea320143a2d349787e918ec40f0b4fb85b6d1" exitCode=0
Oct 30 12:57:40.619318 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.619184    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c898d64f4-tllg2"
Oct 30 12:57:40.619318 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.619228    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c898d64f4-tllg2" event={"ID":"1b563b4e-ff50-4793-a1e8-c4c39f3107c0","Type":"ContainerDied","Data":"9dfaacc058f6b7878be6426f1b9ea320143a2d349787e918ec40f0b4fb85b6d1"}
Oct 30 12:57:40.619318 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.619258    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c898d64f4-tllg2" event={"ID":"1b563b4e-ff50-4793-a1e8-c4c39f3107c0","Type":"ContainerDied","Data":"134e52ce4627db5f61e60353e215d39c76c2f5dc7e8e633e2ad4c8a9db2b7ff6"}
Oct 30 12:57:40.619421 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.619330    5893 scope.go:122] "RemoveContainer" containerID="9dfaacc058f6b7878be6426f1b9ea320143a2d349787e918ec40f0b4fb85b6d1"
Oct 30 12:57:40.628609 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.628588    5893 scope.go:122] "RemoveContainer" containerID="9dfaacc058f6b7878be6426f1b9ea320143a2d349787e918ec40f0b4fb85b6d1"
Oct 30 12:57:40.628895 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:57:40.628873    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dfaacc058f6b7878be6426f1b9ea320143a2d349787e918ec40f0b4fb85b6d1\": container with ID starting with 9dfaacc058f6b7878be6426f1b9ea320143a2d349787e918ec40f0b4fb85b6d1 not found: ID does not exist" containerID="9dfaacc058f6b7878be6426f1b9ea320143a2d349787e918ec40f0b4fb85b6d1"
Oct 30 12:57:40.629005 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.628900    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dfaacc058f6b7878be6426f1b9ea320143a2d349787e918ec40f0b4fb85b6d1"} err="failed to get container status \"9dfaacc058f6b7878be6426f1b9ea320143a2d349787e918ec40f0b4fb85b6d1\": rpc error: code = NotFound desc = could not find container \"9dfaacc058f6b7878be6426f1b9ea320143a2d349787e918ec40f0b4fb85b6d1\": container with ID starting with 9dfaacc058f6b7878be6426f1b9ea320143a2d349787e918ec40f0b4fb85b6d1 not found: ID does not exist"
Oct 30 12:57:40.708436 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.708380    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" podStartSLOduration=1.708367945 podStartE2EDuration="1.708367945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:40.698098555 +0000 UTC m=+702.281528275" watchObservedRunningTime="2026-10-30 12:57:40.708367945 +0000 UTC m=+702.291797637"
Oct 30 12:57:40.748320 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.746899    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c898d64f4-tllg2"]
Oct 30 12:57:40.767306 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:40.765585    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c898d64f4-tllg2"]
Oct 30 12:57:41.587174 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:41.587140    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:41.588863 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:41.588829    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qg8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 12:57:41.589002 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:41.588897    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" podUID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 12:57:42.535296 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:42.533807    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b563b4e-ff50-4793-a1e8-c4c39f3107c0" path="/var/lib/kubelet/pods/1b563b4e-ff50-4793-a1e8-c4c39f3107c0/volumes"
Oct 30 12:57:42.591296 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:42.590797    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qg8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 12:57:42.591296 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:42.590876    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" podUID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 12:57:43.587159 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:43.587123    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qg8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 12:57:43.587685 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:43.587195    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" podUID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 12:57:44.587092 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:44.587059    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qg8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 12:57:44.587266 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:44.587108    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" podUID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 12:57:45.587057 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:45.587018    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qg8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 12:57:45.587254 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:45.587077    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" podUID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 12:57:46.586533 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:46.586499    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qg8wr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Oct 30 12:57:46.587025 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:46.586563    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" podUID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Oct 30 12:57:48.393100 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:48.393066    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:48.393643 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:48.393523    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:48.394517 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:48.394498    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 12:57:48.645048 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:57:48.644967    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:57:54.322158 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.322122    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"]
Oct 30 12:57:54.322784 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.322665    5893 cpu_manager.go:414] "containerMap: removing container" podUID="1b563b4e-ff50-4793-a1e8-c4c39f3107c0" containerName="authorino"
Oct 30 12:57:54.322784 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.322682    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b563b4e-ff50-4793-a1e8-c4c39f3107c0" containerName="authorino"
Oct 30 12:57:54.322784 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.322745    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b563b4e-ff50-4793-a1e8-c4c39f3107c0" containerName="authorino"
Oct 30 12:57:54.326307 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.326264    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.356346 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.356318    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"]
Oct 30 12:57:54.426003 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.425971    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.426161 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.426017    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.426161 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.426090    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d3dc24-762e-4470-a524-c5d7c822ea80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.426238 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.426173    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.426238 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.426197    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.426238 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.426213    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.426379 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.426365    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbrtl\" (UniqueName: \"kubernetes.io/projected/98d3dc24-762e-4470-a524-c5d7c822ea80-kube-api-access-cbrtl\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.426417 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.426393    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.426457 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.426443    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.527571 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.527533    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.527763 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.527602    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.527763 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.527644    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d3dc24-762e-4470-a524-c5d7c822ea80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.527763 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.527689    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.527763 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.527714    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.527763 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.527735    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.528047 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.527829    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbrtl\" (UniqueName: \"kubernetes.io/projected/98d3dc24-762e-4470-a524-c5d7c822ea80-kube-api-access-cbrtl\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.528047 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.527865    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.528047 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.527931    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.528211 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.528104    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.528211 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.528119    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.528360 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.528335    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.528591 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.528565    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.528718 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.528612    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d3dc24-762e-4470-a524-c5d7c822ea80-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.530301 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.530253    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.530407 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.530308    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.549881 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.549859    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.550009 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.549992    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbrtl\" (UniqueName: \"kubernetes.io/projected/98d3dc24-762e-4470-a524-c5d7c822ea80-kube-api-access-cbrtl\") pod \"maas-default-gateway-openshift-default-58c6585447-5tgdl\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:54.638782 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:54.638703    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:55.003910 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.003863    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"]
Oct 30 12:57:55.006864 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:57:55.006830    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98d3dc24_762e_4470_a524_c5d7c822ea80.slice/crio-65a9c20c3b38676e7b2d1c3b7f55e92aca7ee7edd411c120924d2dc86ffb8976 WatchSource:0}: Error finding container 65a9c20c3b38676e7b2d1c3b7f55e92aca7ee7edd411c120924d2dc86ffb8976: Status 404 returned error can't find the container with id 65a9c20c3b38676e7b2d1c3b7f55e92aca7ee7edd411c120924d2dc86ffb8976
Oct 30 12:57:55.009057 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.009023    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:55.009139 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.009089    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:55.009139 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.009117    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:57:55.508355 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.508320    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-69c996b8fd-7phsv"]
Oct 30 12:57:55.512158 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.512139    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-69c996b8fd-7phsv"
Oct 30 12:57:55.527676 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.527652    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 12:57:55.538104 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.538082    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-gbh2n\""
Oct 30 12:57:55.538225 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.538187    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 12:57:55.568221 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.568182    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-69c996b8fd-7phsv"]
Oct 30 12:57:55.640150 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.640119    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn4gw\" (UniqueName: \"kubernetes.io/projected/dca88d7b-ad83-4b4c-bded-ced88f047b8b-kube-api-access-zn4gw\") pod \"ai-gateway-controller-69c996b8fd-7phsv\" (UID: \"dca88d7b-ad83-4b4c-bded-ced88f047b8b\") " pod="opendatahub/ai-gateway-controller-69c996b8fd-7phsv"
Oct 30 12:57:55.640373 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.640199    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca88d7b-ad83-4b4c-bded-ced88f047b8b-metrics-certs\") pod \"ai-gateway-controller-69c996b8fd-7phsv\" (UID: \"dca88d7b-ad83-4b4c-bded-ced88f047b8b\") " pod="opendatahub/ai-gateway-controller-69c996b8fd-7phsv"
Oct 30 12:57:55.726244 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.726212    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" event={"ID":"98d3dc24-762e-4470-a524-c5d7c822ea80","Type":"ContainerStarted","Data":"5ba9225218b01720724f795a15a11d88be7856e25333d61294a7e1f45c3743eb"}
Oct 30 12:57:55.726436 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.726259    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" event={"ID":"98d3dc24-762e-4470-a524-c5d7c822ea80","Type":"ContainerStarted","Data":"65a9c20c3b38676e7b2d1c3b7f55e92aca7ee7edd411c120924d2dc86ffb8976"}
Oct 30 12:57:55.760351 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.749847    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn4gw\" (UniqueName: \"kubernetes.io/projected/dca88d7b-ad83-4b4c-bded-ced88f047b8b-kube-api-access-zn4gw\") pod \"ai-gateway-controller-69c996b8fd-7phsv\" (UID: \"dca88d7b-ad83-4b4c-bded-ced88f047b8b\") " pod="opendatahub/ai-gateway-controller-69c996b8fd-7phsv"
Oct 30 12:57:55.760351 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.749958    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca88d7b-ad83-4b4c-bded-ced88f047b8b-metrics-certs\") pod \"ai-gateway-controller-69c996b8fd-7phsv\" (UID: \"dca88d7b-ad83-4b4c-bded-ced88f047b8b\") " pod="opendatahub/ai-gateway-controller-69c996b8fd-7phsv"
Oct 30 12:57:55.766026 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.765992    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dca88d7b-ad83-4b4c-bded-ced88f047b8b-metrics-certs\") pod \"ai-gateway-controller-69c996b8fd-7phsv\" (UID: \"dca88d7b-ad83-4b4c-bded-ced88f047b8b\") " pod="opendatahub/ai-gateway-controller-69c996b8fd-7phsv"
Oct 30 12:57:55.769095 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.769045    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" podStartSLOduration=1.76903192 podStartE2EDuration="1.76903192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:55.767789028 +0000 UTC m=+717.351218747" watchObservedRunningTime="2026-10-30 12:57:55.76903192 +0000 UTC m=+717.352461612"
Oct 30 12:57:55.784870 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.784845    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn4gw\" (UniqueName: \"kubernetes.io/projected/dca88d7b-ad83-4b4c-bded-ced88f047b8b-kube-api-access-zn4gw\") pod \"ai-gateway-controller-69c996b8fd-7phsv\" (UID: \"dca88d7b-ad83-4b4c-bded-ced88f047b8b\") " pod="opendatahub/ai-gateway-controller-69c996b8fd-7phsv"
Oct 30 12:57:55.824919 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:55.824882    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-69c996b8fd-7phsv"
Oct 30 12:57:56.152970 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:56.152880    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-69c996b8fd-7phsv"]
Oct 30 12:57:56.167612 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:57:56.167574    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca88d7b_ad83_4b4c_bded_ced88f047b8b.slice/crio-3684ba05eeac2107615d652a15d58bd19ccaa31e3643887a687489434381807c WatchSource:0}: Error finding container 3684ba05eeac2107615d652a15d58bd19ccaa31e3643887a687489434381807c: Status 404 returned error can't find the container with id 3684ba05eeac2107615d652a15d58bd19ccaa31e3643887a687489434381807c
Oct 30 12:57:56.639598 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:56.639520    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:57:56.665793 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:56.665432    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5tgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:57:56.665793 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:56.665495    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" podUID="98d3dc24-762e-4470-a524-c5d7c822ea80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 12:57:56.735042 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:56.735004    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-69c996b8fd-7phsv" event={"ID":"dca88d7b-ad83-4b4c-bded-ced88f047b8b","Type":"ContainerStarted","Data":"3684ba05eeac2107615d652a15d58bd19ccaa31e3643887a687489434381807c"}
Oct 30 12:57:57.639520 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:57.639485    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5tgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:57:57.639697 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:57.639553    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" podUID="98d3dc24-762e-4470-a524-c5d7c822ea80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 12:57:58.639609 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:58.639197    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5tgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:57:58.639609 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:58.639257    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" podUID="98d3dc24-762e-4470-a524-c5d7c822ea80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 12:57:58.983141 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:58.983118    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 12:57:59.639234 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:59.639199    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5tgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:57:59.639428 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:59.639251    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" podUID="98d3dc24-762e-4470-a524-c5d7c822ea80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 12:57:59.776713 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:59.776674    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-69c996b8fd-7phsv" event={"ID":"dca88d7b-ad83-4b4c-bded-ced88f047b8b","Type":"ContainerStarted","Data":"407feded7fcffabc1d404905690660be133618d390668df16e65fa4e6615c0e9"}
Oct 30 12:57:59.777105 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:59.776807    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-69c996b8fd-7phsv"
Oct 30 12:57:59.815194 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:57:59.815147    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-69c996b8fd-7phsv" podStartSLOduration=2.013986374 podStartE2EDuration="4.815133721s" totalImagesPullingTime="2.801147347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:57:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:57:59.81336519 +0000 UTC m=+721.396794899" watchObservedRunningTime="2026-10-30 12:57:59.815133721 +0000 UTC m=+721.398563448"
Oct 30 12:58:00.639239 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:00.639205    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5tgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:58:00.639410 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:00.639257    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" podUID="98d3dc24-762e-4470-a524-c5d7c822ea80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 12:58:01.640134 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:01.640097    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5tgdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Oct 30 12:58:01.640545 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:01.640156    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" podUID="98d3dc24-762e-4470-a524-c5d7c822ea80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Oct 30 12:58:03.281021 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:03.280989    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:58:03.281539 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:03.281413    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:58:03.282181 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:03.282152    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 12:58:10.784329 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:10.784243    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-69c996b8fd-7phsv"
Oct 30 12:58:18.644168 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:58:18.644131    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:58:38.606058 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.606027    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"]
Oct 30 12:58:38.612404 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.612381    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.632818 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.632792    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-09db16-openshift-default-dockercfg-l5hf4\""
Oct 30 12:58:38.650086 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.650057    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"]
Oct 30 12:58:38.691715 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.691683    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.691900 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.691742    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.691900 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.691873    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.692018 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.691929    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.692018 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.691963    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.692098 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.692078    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac623bd1-d3de-47f4-9e06-a2261a160df8-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.692134 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.692112    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.692166 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.692136    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94ztk\" (UniqueName: \"kubernetes.io/projected/ac623bd1-d3de-47f4-9e06-a2261a160df8-kube-api-access-94ztk\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.692257 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.692238    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.793340 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.793304    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94ztk\" (UniqueName: \"kubernetes.io/projected/ac623bd1-d3de-47f4-9e06-a2261a160df8-kube-api-access-94ztk\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.793529 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.793377    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.793529 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.793439    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.793529 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.793483    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.793529 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.793517    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.793749 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.793559    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.793749 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.793588    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.793749 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.793644    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac623bd1-d3de-47f4-9e06-a2261a160df8-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.793749 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.793674    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.794015 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.793992    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-data\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.794074 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.794005    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-workload-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.794297 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.794253    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-credential-socket\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.794477 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.794301    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-workload-certs\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.795416 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.795387    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac623bd1-d3de-47f4-9e06-a2261a160df8-istiod-ca-cert\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.797630 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.797603    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-podinfo\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.798489 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.798468    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-envoy\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.832111 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.832052    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-token\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.832111 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.832090    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94ztk\" (UniqueName: \"kubernetes.io/projected/ac623bd1-d3de-47f4-9e06-a2261a160df8-kube-api-access-94ztk\") pod \"e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") " pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:38.926730 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:38.926655    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:39.153739 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.153705    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"]
Oct 30 12:58:39.158197 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.158175    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.182921 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.182825    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"]
Oct 30 12:58:39.198263 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.198238    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-credential-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.198428 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.198307    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-istio-data\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.198428 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.198362    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3873ae03-39a5-4f72-924d-6eba73017d74-istio-podinfo\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.198515 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.198456    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3873ae03-39a5-4f72-924d-6eba73017d74-istio-token\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.198515 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.198484    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-istio-envoy\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.198620 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.198542    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-workload-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.198684 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.198661    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3873ae03-39a5-4f72-924d-6eba73017d74-istiod-ca-cert\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.198740 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.198721    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvd2v\" (UniqueName: \"kubernetes.io/projected/3873ae03-39a5-4f72-924d-6eba73017d74-kube-api-access-fvd2v\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.198797 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.198775    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-workload-certs\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.231935 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.231908    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c2ce75-openshift-default-dockercfg-44wkt\""
Oct 30 12:58:39.259070 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.259039    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"]
Oct 30 12:58:39.264221 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:58:39.264191    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac623bd1_d3de_47f4_9e06_a2261a160df8.slice/crio-21fed8a66b85a6b7b5636ee1b455f82943bbe30b8f163ca76d5ac4f1e5bbcb65 WatchSource:0}: Error finding container 21fed8a66b85a6b7b5636ee1b455f82943bbe30b8f163ca76d5ac4f1e5bbcb65: Status 404 returned error can't find the container with id 21fed8a66b85a6b7b5636ee1b455f82943bbe30b8f163ca76d5ac4f1e5bbcb65
Oct 30 12:58:39.266477 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.266447    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:39.266568 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.266515    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:39.266568 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.266547    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:39.299405 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.299382    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-workload-certs\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.299537 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.299418    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-credential-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.299537 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.299465    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-istio-data\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.299537 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.299484    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3873ae03-39a5-4f72-924d-6eba73017d74-istio-podinfo\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.300246 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.299593    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3873ae03-39a5-4f72-924d-6eba73017d74-istio-token\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.300246 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.299644    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-istio-envoy\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.300246 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.299740    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-workload-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.300246 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.299786    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3873ae03-39a5-4f72-924d-6eba73017d74-istiod-ca-cert\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.300246 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.299813    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-workload-certs\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.300246 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.299836    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvd2v\" (UniqueName: \"kubernetes.io/projected/3873ae03-39a5-4f72-924d-6eba73017d74-kube-api-access-fvd2v\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.300246 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.299878    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-istio-data\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.300608 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.300363    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-credential-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.300705 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.300689    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-workload-socket\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.300832 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.300808    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3873ae03-39a5-4f72-924d-6eba73017d74-istiod-ca-cert\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.302739 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.302714    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-istio-envoy\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.302739 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.302724    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3873ae03-39a5-4f72-924d-6eba73017d74-istio-podinfo\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.332597 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.332563    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvd2v\" (UniqueName: \"kubernetes.io/projected/3873ae03-39a5-4f72-924d-6eba73017d74-kube-api-access-fvd2v\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.332730 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.332628    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3873ae03-39a5-4f72-924d-6eba73017d74-istio-token\") pod \"e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") " pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.476197 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.476164    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:39.487953 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.487924    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"]
Oct 30 12:58:39.493826 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.493802    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.512379 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.512350    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-8b46c9-openshift-default-dockercfg-9flrx\""
Oct 30 12:58:39.535157 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.535127    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"]
Oct 30 12:58:39.603224 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.603194    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.603501 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.603244    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.603501 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.603367    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.603666 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.603550    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.603666 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.603608    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.603757 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.603687    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.603757 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.603739    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhzh4\" (UniqueName: \"kubernetes.io/projected/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-kube-api-access-bhzh4\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.603829 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.603762    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.604423 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.604394    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.707903 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.707859    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.708471 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.707949    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.708471 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.707992    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.708471 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.708050    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.708471 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.708095    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.708471 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.708159    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.708471 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.708195    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhzh4\" (UniqueName: \"kubernetes.io/projected/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-kube-api-access-bhzh4\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.708471 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.708221    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.708471 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.708266    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.708471 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.708347    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.709300 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.708616    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.709300 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.708717    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.709300 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.709247    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.709543 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.709348    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.714702 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.714608    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.715151 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.715076    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.742366 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.742234    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.742667 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.742644    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhzh4\" (UniqueName: \"kubernetes.io/projected/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-kube-api-access-bhzh4\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.811969 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.811937    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:39.827105 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.827077    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"]
Oct 30 12:58:39.831491 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:58:39.831459    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3873ae03_39a5_4f72_924d_6eba73017d74.slice/crio-5a358995f488744787d7438b68acf0a53d356a0563d3e338ebeb49726d412ebb WatchSource:0}: Error finding container 5a358995f488744787d7438b68acf0a53d356a0563d3e338ebeb49726d412ebb: Status 404 returned error can't find the container with id 5a358995f488744787d7438b68acf0a53d356a0563d3e338ebeb49726d412ebb
Oct 30 12:58:39.834083 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.834051    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:39.834188 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.834114    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:39.834188 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.834140    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:39.965615 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.965584    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" event={"ID":"3873ae03-39a5-4f72-924d-6eba73017d74","Type":"ContainerStarted","Data":"436ec0a82c043c498497aaca1b3c15a6985f5696e4c78cbb2e4b495c8a804fde"}
Oct 30 12:58:39.965771 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.965645    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" event={"ID":"3873ae03-39a5-4f72-924d-6eba73017d74","Type":"ContainerStarted","Data":"5a358995f488744787d7438b68acf0a53d356a0563d3e338ebeb49726d412ebb"}
Oct 30 12:58:39.967841 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.967814    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4" event={"ID":"ac623bd1-d3de-47f4-9e06-a2261a160df8","Type":"ContainerStarted","Data":"bfdc6246ef9576635f40aa600abe74f364919f6ff842637934def3dce6d5612c"}
Oct 30 12:58:39.967974 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:39.967864    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4" event={"ID":"ac623bd1-d3de-47f4-9e06-a2261a160df8","Type":"ContainerStarted","Data":"21fed8a66b85a6b7b5636ee1b455f82943bbe30b8f163ca76d5ac4f1e5bbcb65"}
Oct 30 12:58:40.061967 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.061856    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" podStartSLOduration=1.061839388 podStartE2EDuration="1.061839388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:40.060217182 +0000 UTC m=+761.643646878" watchObservedRunningTime="2026-10-30 12:58:40.061839388 +0000 UTC m=+761.645269078"
Oct 30 12:58:40.135603 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.135556    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4" podStartSLOduration=2.135541156 podStartE2EDuration="2.135541156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:40.132894823 +0000 UTC m=+761.716324517" watchObservedRunningTime="2026-10-30 12:58:40.135541156 +0000 UTC m=+761.718970879"
Oct 30 12:58:40.170067 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.170037    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"]
Oct 30 12:58:40.174576 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:58:40.174546    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1b30da_f0f8_4eb0_b50b_1d2ae668801b.slice/crio-822d8fb8efb73e0875ae708c2e98cafc2123b9b64a4605aedcd517ef592fa7cf WatchSource:0}: Error finding container 822d8fb8efb73e0875ae708c2e98cafc2123b9b64a4605aedcd517ef592fa7cf: Status 404 returned error can't find the container with id 822d8fb8efb73e0875ae708c2e98cafc2123b9b64a4605aedcd517ef592fa7cf
Oct 30 12:58:40.177078 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.177038    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:40.177207 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.177128    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:40.177207 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.177172    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:58:40.476593 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.476560    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:40.482644 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.482619    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:40.927972 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.927902    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:40.932694 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.932670    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:40.973917 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.973876    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87" event={"ID":"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b","Type":"ContainerStarted","Data":"940dfc9a35b250df4ee1698effe2d55299527c217dcd71b8c60c271745b5d810"}
Oct 30 12:58:40.974063 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.973939    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87" event={"ID":"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b","Type":"ContainerStarted","Data":"822d8fb8efb73e0875ae708c2e98cafc2123b9b64a4605aedcd517ef592fa7cf"}
Oct 30 12:58:40.974280 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.974251    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:40.974359 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.974328    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:40.975084 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.975056    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:58:40.975252 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:40.975242    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:58:41.126667 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:41.126614    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87" podStartSLOduration=2.126602968 podStartE2EDuration="2.126602968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:41.126466551 +0000 UTC m=+762.709896266" watchObservedRunningTime="2026-10-30 12:58:41.126602968 +0000 UTC m=+762.710032660"
Oct 30 12:58:41.812840 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:41.812807    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:41.820610 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:41.820584    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:41.980243 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:41.980207    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:41.980871 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:41.980852    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:58:45.361998 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:45.361955    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"]
Oct 30 12:58:45.368254 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:45.368209    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"
Oct 30 12:58:45.394872 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:45.394840    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"]
Oct 30 12:58:45.408879 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:45.408845    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-09db16-dockercfg-mjg7j\""
Oct 30 12:58:45.409082 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:45.408932    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-09db16\""
Oct 30 12:58:45.472263 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:45.472227    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9afa66e-f49c-4afb-a25f-928ec672683c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9\" (UID: \"d9afa66e-f49c-4afb-a25f-928ec672683c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"
Oct 30 12:58:45.472447 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:45.472406    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mfcj\" (UniqueName: \"kubernetes.io/projected/d9afa66e-f49c-4afb-a25f-928ec672683c-kube-api-access-8mfcj\") pod \"payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9\" (UID: \"d9afa66e-f49c-4afb-a25f-928ec672683c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"
Oct 30 12:58:45.573048 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:45.573017    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mfcj\" (UniqueName: \"kubernetes.io/projected/d9afa66e-f49c-4afb-a25f-928ec672683c-kube-api-access-8mfcj\") pod \"payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9\" (UID: \"d9afa66e-f49c-4afb-a25f-928ec672683c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"
Oct 30 12:58:45.573228 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:45.573100    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9afa66e-f49c-4afb-a25f-928ec672683c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9\" (UID: \"d9afa66e-f49c-4afb-a25f-928ec672683c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"
Oct 30 12:58:45.573953 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:45.573933    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9afa66e-f49c-4afb-a25f-928ec672683c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9\" (UID: \"d9afa66e-f49c-4afb-a25f-928ec672683c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"
Oct 30 12:58:45.604022 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:45.603994    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mfcj\" (UniqueName: \"kubernetes.io/projected/d9afa66e-f49c-4afb-a25f-928ec672683c-kube-api-access-8mfcj\") pod \"payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9\" (UID: \"d9afa66e-f49c-4afb-a25f-928ec672683c\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"
Oct 30 12:58:45.686935 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:45.686856    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"
Oct 30 12:58:45.988743 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:45.988711    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"]
Oct 30 12:58:45.992927 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:58:45.992901    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9afa66e_f49c_4afb_a25f_928ec672683c.slice/crio-899e8104f88cceb78bd68260186cb0c8219b512e0e50083677af8adadd9674dc WatchSource:0}: Error finding container 899e8104f88cceb78bd68260186cb0c8219b512e0e50083677af8adadd9674dc: Status 404 returned error can't find the container with id 899e8104f88cceb78bd68260186cb0c8219b512e0e50083677af8adadd9674dc
Oct 30 12:58:46.003076 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.003043    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9" event={"ID":"d9afa66e-f49c-4afb-a25f-928ec672683c","Type":"ContainerStarted","Data":"899e8104f88cceb78bd68260186cb0c8219b512e0e50083677af8adadd9674dc"}
Oct 30 12:58:46.747388 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.747353    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"]
Oct 30 12:58:46.751355 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.751335    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:46.769929 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.769903    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-c90b9e-serving-cert\""
Oct 30 12:58:46.785467 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.785434    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"]
Oct 30 12:58:46.785884 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.785850    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:46.786024 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.785905    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:46.786024 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.786010    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-metrics-certs\") pod \"maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:46.786138 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.786038    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpjkz\" (UniqueName: \"kubernetes.io/projected/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-kube-api-access-jpjkz\") pod \"maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:46.886943 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.886909    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-metrics-certs\") pod \"maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:46.887138 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.886958    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpjkz\" (UniqueName: \"kubernetes.io/projected/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-kube-api-access-jpjkz\") pod \"maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:46.887138 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.887098    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:46.887138 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.887129    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:46.887505 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:58:46.887327    5893 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-serving-cert: secret "maas-api-e2e-worker-w0-c90b9e-serving-cert" not found
Oct 30 12:58:46.887505 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:58:46.887397    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-maas-api-tls podName:df8bcf75-23e1-4a1c-9384-6781cbdf3d38 nodeName:}" failed. No retries permitted until 2026-10-30 12:58:47.387379736 +0000 UTC m=+768.970809407 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-maas-api-tls") pod "maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx" (UID: "df8bcf75-23e1-4a1c-9384-6781cbdf3d38") : secret "maas-api-e2e-worker-w0-c90b9e-serving-cert" not found
Oct 30 12:58:46.887958 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.887940    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:46.889959 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.889937    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-metrics-certs\") pod \"maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:46.910028 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:46.909998    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpjkz\" (UniqueName: \"kubernetes.io/projected/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-kube-api-access-jpjkz\") pod \"maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:47.395436 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:47.394752    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:47.398805 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:47.398770    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:47.667949 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:47.667864    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:48.056655 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:48.056626    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"]
Oct 30 12:58:48.647791 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:58:48.647756    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:58:49.023622 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.023584    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx" event={"ID":"df8bcf75-23e1-4a1c-9384-6781cbdf3d38","Type":"ContainerStarted","Data":"e8b24e381a99099b89ae342eddaf923522db307d99629731adb5d95599e06ce0"}
Oct 30 12:58:49.023836 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.023655    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx" event={"ID":"df8bcf75-23e1-4a1c-9384-6781cbdf3d38","Type":"ContainerStarted","Data":"eb353d26ae57aade4552768025aaba289999f59bf20dc6f95bbc4a47d36e7e1c"}
Oct 30 12:58:49.023836 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.023688    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:49.024054 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.024037    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"]
Oct 30 12:58:49.027974 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.027956    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"
Oct 30 12:58:49.028104 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.027981    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9" event={"ID":"d9afa66e-f49c-4afb-a25f-928ec672683c","Type":"ContainerStarted","Data":"727d94a13ad8ad2c779a17201d1a5acc91311c430b5c38a996c6bbe78d9f4f95"}
Oct 30 12:58:49.028104 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.028058    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:49.047361 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.047332    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-dd90cb-serving-cert\""
Oct 30 12:58:49.055542 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.055514    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"]
Oct 30 12:58:49.067466 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.067409    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx" podStartSLOduration=2.694320928 podStartE2EDuration="3.067393611s" totalImagesPullingTime="373.072683ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:49.064754092 +0000 UTC m=+770.648183792" watchObservedRunningTime="2026-10-30 12:58:49.067393611 +0000 UTC m=+770.650823304"
Oct 30 12:58:49.115317 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.115250    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1529e3f4-294a-4f88-a8e2-438b5d0778c6-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:49.115516 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.115360    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1529e3f4-294a-4f88-a8e2-438b5d0778c6-maas-api-tls\") pod \"maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:49.115516 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.115426    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8z9d\" (UniqueName: \"kubernetes.io/projected/1529e3f4-294a-4f88-a8e2-438b5d0778c6-kube-api-access-s8z9d\") pod \"maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:49.115516 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.115478    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1529e3f4-294a-4f88-a8e2-438b5d0778c6-metrics-certs\") pod \"maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:49.155339 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.155260    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9" podStartSLOduration=2.045198833 podStartE2EDuration="4.155240956s" totalImagesPullingTime="2.110042123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:49.15436849 +0000 UTC m=+770.737798184" watchObservedRunningTime="2026-10-30 12:58:49.155240956 +0000 UTC m=+770.738670651"
Oct 30 12:58:49.216346 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.216314    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8z9d\" (UniqueName: \"kubernetes.io/projected/1529e3f4-294a-4f88-a8e2-438b5d0778c6-kube-api-access-s8z9d\") pod \"maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:49.216539 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.216390    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1529e3f4-294a-4f88-a8e2-438b5d0778c6-metrics-certs\") pod \"maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:49.216609 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.216542    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1529e3f4-294a-4f88-a8e2-438b5d0778c6-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:49.216609 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.216572    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1529e3f4-294a-4f88-a8e2-438b5d0778c6-maas-api-tls\") pod \"maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:49.217517 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.217490    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1529e3f4-294a-4f88-a8e2-438b5d0778c6-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:49.219004 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.218984    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1529e3f4-294a-4f88-a8e2-438b5d0778c6-maas-api-tls\") pod \"maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:49.219083 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.219049    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1529e3f4-294a-4f88-a8e2-438b5d0778c6-metrics-certs\") pod \"maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:49.238465 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.238440    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8z9d\" (UniqueName: \"kubernetes.io/projected/1529e3f4-294a-4f88-a8e2-438b5d0778c6-kube-api-access-s8z9d\") pod \"maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:49.344117 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.344026    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:49.657752 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:49.657660    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"]
Oct 30 12:58:49.662035 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:58:49.662010    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1529e3f4_294a_4f88_a8e2_438b5d0778c6.slice/crio-b5270b712f5d1c310733e5827eade6bcc54279e59703617e1a3d94b9f42c91f8 WatchSource:0}: Error finding container b5270b712f5d1c310733e5827eade6bcc54279e59703617e1a3d94b9f42c91f8: Status 404 returned error can't find the container with id b5270b712f5d1c310733e5827eade6bcc54279e59703617e1a3d94b9f42c91f8
Oct 30 12:58:50.032394 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:50.032357    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4" event={"ID":"1529e3f4-294a-4f88-a8e2-438b5d0778c6","Type":"ContainerStarted","Data":"f61cb449505b3a88769c94fdd9fc3cc462877dd065319c1fe71d6bdac2979f0d"}
Oct 30 12:58:50.032603 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:50.032420    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4" event={"ID":"1529e3f4-294a-4f88-a8e2-438b5d0778c6","Type":"ContainerStarted","Data":"b5270b712f5d1c310733e5827eade6bcc54279e59703617e1a3d94b9f42c91f8"}
Oct 30 12:58:50.032713 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:50.032684    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:51.039232 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.039183    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4" podStartSLOduration=2.772746905 podStartE2EDuration="3.039168137s" totalImagesPullingTime="266.421232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:50.073706194 +0000 UTC m=+771.657135890" watchObservedRunningTime="2026-10-30 12:58:51.039168137 +0000 UTC m=+772.622597830"
Oct 30 12:58:51.040472 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.040447    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"]
Oct 30 12:58:51.044629 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.044609    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:51.057814 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.057791    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-7a3603-serving-cert\""
Oct 30 12:58:51.072824 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.072791    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"]
Oct 30 12:58:51.141746 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.141706    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d84ed3c0-d5fc-4961-a037-1c20a8704e50-metrics-certs\") pod \"maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:51.141929 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.141855    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c95f5\" (UniqueName: \"kubernetes.io/projected/d84ed3c0-d5fc-4961-a037-1c20a8704e50-kube-api-access-c95f5\") pod \"maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:51.141929 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.141894    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84ed3c0-d5fc-4961-a037-1c20a8704e50-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:51.142042 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.141929    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d84ed3c0-d5fc-4961-a037-1c20a8704e50-maas-api-tls\") pod \"maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:51.243359 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.243326    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d84ed3c0-d5fc-4961-a037-1c20a8704e50-metrics-certs\") pod \"maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:51.243601 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.243397    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c95f5\" (UniqueName: \"kubernetes.io/projected/d84ed3c0-d5fc-4961-a037-1c20a8704e50-kube-api-access-c95f5\") pod \"maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:51.243601 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.243435    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84ed3c0-d5fc-4961-a037-1c20a8704e50-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:51.243601 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.243472    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d84ed3c0-d5fc-4961-a037-1c20a8704e50-maas-api-tls\") pod \"maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:51.244598 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.244574    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84ed3c0-d5fc-4961-a037-1c20a8704e50-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:51.246493 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.246471    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d84ed3c0-d5fc-4961-a037-1c20a8704e50-maas-api-tls\") pod \"maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:51.246616 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.246511    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d84ed3c0-d5fc-4961-a037-1c20a8704e50-metrics-certs\") pod \"maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:51.265530 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.265504    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c95f5\" (UniqueName: \"kubernetes.io/projected/d84ed3c0-d5fc-4961-a037-1c20a8704e50-kube-api-access-c95f5\") pod \"maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:51.359102 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.359000    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:51.686944 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:51.679801    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"]
Oct 30 12:58:52.046735 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:52.046707    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98" event={"ID":"d84ed3c0-d5fc-4961-a037-1c20a8704e50","Type":"ContainerStarted","Data":"5ef024fd02f2983440d15ece989469783db45a66ec4b104e42f319aa28f61604"}
Oct 30 12:58:53.040082 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.040049    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"]
Oct 30 12:58:53.043672 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.043650    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:53.053685 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.053656    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98" event={"ID":"d84ed3c0-d5fc-4961-a037-1c20a8704e50","Type":"ContainerStarted","Data":"a68bf5abd0d38c39d51558cc09bd88847e52c9f18b4c1d9b947dbf91833148f7"}
Oct 30 12:58:53.054032 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.053805    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:53.061255 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.061233    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-09db16-serving-cert\""
Oct 30 12:58:53.082523 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.082487    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"]
Oct 30 12:58:53.156609 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.156547    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98" podStartSLOduration=1.832784582 podStartE2EDuration="2.156531789s" totalImagesPullingTime="323.747207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:53.155830234 +0000 UTC m=+774.739259940" watchObservedRunningTime="2026-10-30 12:58:53.156531789 +0000 UTC m=+774.739961485"
Oct 30 12:58:53.167436 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.167403    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75c100c-a950-4e30-9c47-51ec63a8b09a-metrics-certs\") pod \"maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:53.167612 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.167476    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75c100c-a950-4e30-9c47-51ec63a8b09a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:53.167612 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.167594    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75c100c-a950-4e30-9c47-51ec63a8b09a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:53.167763 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.167741    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2td8l\" (UniqueName: \"kubernetes.io/projected/b75c100c-a950-4e30-9c47-51ec63a8b09a-kube-api-access-2td8l\") pod \"maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:53.269350 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.269320    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75c100c-a950-4e30-9c47-51ec63a8b09a-metrics-certs\") pod \"maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:53.269556 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.269389    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75c100c-a950-4e30-9c47-51ec63a8b09a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:53.269556 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.269415    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75c100c-a950-4e30-9c47-51ec63a8b09a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:53.269556 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.269477    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2td8l\" (UniqueName: \"kubernetes.io/projected/b75c100c-a950-4e30-9c47-51ec63a8b09a-kube-api-access-2td8l\") pod \"maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:53.270250 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.270170    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75c100c-a950-4e30-9c47-51ec63a8b09a-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:53.272115 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.272095    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75c100c-a950-4e30-9c47-51ec63a8b09a-metrics-certs\") pod \"maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:53.272204 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.272116    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75c100c-a950-4e30-9c47-51ec63a8b09a-maas-api-tls\") pod \"maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:53.292652 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.292602    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2td8l\" (UniqueName: \"kubernetes.io/projected/b75c100c-a950-4e30-9c47-51ec63a8b09a-kube-api-access-2td8l\") pod \"maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:53.365601 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.364746    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:53.672469 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:53.672378    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"]
Oct 30 12:58:53.676728 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:58:53.676706    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75c100c_a950_4e30_9c47_51ec63a8b09a.slice/crio-ed6a8e8242bd0f82792c6f9f7f0c278560e34f92a00495210485fc07dadbdf43 WatchSource:0}: Error finding container ed6a8e8242bd0f82792c6f9f7f0c278560e34f92a00495210485fc07dadbdf43: Status 404 returned error can't find the container with id ed6a8e8242bd0f82792c6f9f7f0c278560e34f92a00495210485fc07dadbdf43
Oct 30 12:58:54.060085 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:54.060051    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6" event={"ID":"b75c100c-a950-4e30-9c47-51ec63a8b09a","Type":"ContainerStarted","Data":"ed6a8e8242bd0f82792c6f9f7f0c278560e34f92a00495210485fc07dadbdf43"}
Oct 30 12:58:55.038546 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:55.038519    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 12:58:55.066655 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:55.066621    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6" event={"ID":"b75c100c-a950-4e30-9c47-51ec63a8b09a","Type":"ContainerStarted","Data":"6c4baa5ca1c1ef7e35ce9a25cc1e284c7034c5b3e92287125b1adf8e945546a5"}
Oct 30 12:58:55.067021 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:55.066687    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:58:55.136123 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:55.136063    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6" podStartSLOduration=1.727946625 podStartE2EDuration="2.136048706s" totalImagesPullingTime="408.102081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:58:55.134570777 +0000 UTC m=+776.718000470" watchObservedRunningTime="2026-10-30 12:58:55.136048706 +0000 UTC m=+776.719478396"
Oct 30 12:58:56.065568 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:56.065524    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 12:58:57.510189 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.510158    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"]
Oct 30 12:58:57.514230 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.514209    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:57.527120 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.527096    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-c2ce75-serving-cert\""
Oct 30 12:58:57.561298 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.540338    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"]
Oct 30 12:58:57.632548 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.632514    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-metrics-certs\") pod \"maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:57.632548 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.632551    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:57.632747 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.632628    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b215f236-b763-4a75-be1c-0edeb29aaae9-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:57.632783 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.632765    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcrxw\" (UniqueName: \"kubernetes.io/projected/b215f236-b763-4a75-be1c-0edeb29aaae9-kube-api-access-zcrxw\") pod \"maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:57.734072 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.734034    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-metrics-certs\") pod \"maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:57.734072 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.734073    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:57.734323 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.734102    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b215f236-b763-4a75-be1c-0edeb29aaae9-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:57.734323 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.734152    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcrxw\" (UniqueName: \"kubernetes.io/projected/b215f236-b763-4a75-be1c-0edeb29aaae9-kube-api-access-zcrxw\") pod \"maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:57.734323 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:58:57.734204    5893 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-serving-cert: secret "maas-api-e2e-worker-w2-c2ce75-serving-cert" not found
Oct 30 12:58:57.734323 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:58:57.734318    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-maas-api-tls podName:b215f236-b763-4a75-be1c-0edeb29aaae9 nodeName:}" failed. No retries permitted until 2026-10-30 12:58:58.234255707 +0000 UTC m=+779.817685395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-maas-api-tls") pod "maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h" (UID: "b215f236-b763-4a75-be1c-0edeb29aaae9") : secret "maas-api-e2e-worker-w2-c2ce75-serving-cert" not found
Oct 30 12:58:57.734833 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.734815    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b215f236-b763-4a75-be1c-0edeb29aaae9-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:57.736767 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.736749    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-metrics-certs\") pod \"maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:57.768820 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:57.768741    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcrxw\" (UniqueName: \"kubernetes.io/projected/b215f236-b763-4a75-be1c-0edeb29aaae9-kube-api-access-zcrxw\") pod \"maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:58.238642 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:58.238607    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:58.266916 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:58.256122    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:58.430785 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:58.430755    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:58:58.768397 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:58.765463    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"]
Oct 30 12:58:58.773134 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:58:58.773098    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb215f236_b763_4a75_be1c_0edeb29aaae9.slice/crio-36ce9564bedb5fe3ddb861928c126efa4633ec6533a818d61130bfb42e7e9a75 WatchSource:0}: Error finding container 36ce9564bedb5fe3ddb861928c126efa4633ec6533a818d61130bfb42e7e9a75: Status 404 returned error can't find the container with id 36ce9564bedb5fe3ddb861928c126efa4633ec6533a818d61130bfb42e7e9a75
Oct 30 12:58:59.083455 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:59.082626    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 12:58:59.132905 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:58:59.132878    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h" event={"ID":"b215f236-b763-4a75-be1c-0edeb29aaae9","Type":"ContainerStarted","Data":"36ce9564bedb5fe3ddb861928c126efa4633ec6533a818d61130bfb42e7e9a75"}
Oct 30 12:59:00.036078 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:00.036053    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"
Oct 30 12:59:00.167723 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:00.167665    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h" event={"ID":"b215f236-b763-4a75-be1c-0edeb29aaae9","Type":"ContainerStarted","Data":"8016c30d1fbef9d21f64a54a76f9f949479bf7531014ca9472e545831b9cb1f8"}
Oct 30 12:59:00.168917 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:00.168894    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:59:00.209560 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:00.209512    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h" podStartSLOduration=2.88493972 podStartE2EDuration="3.209497892s" totalImagesPullingTime="324.558172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:58:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:00.208194592 +0000 UTC m=+781.791624310" watchObservedRunningTime="2026-10-30 12:59:00.209497892 +0000 UTC m=+781.792927584"
Oct 30 12:59:01.122325 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:01.122266    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 12:59:07.190186 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:07.190102    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 12:59:07.214948 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:07.214911    5893 generic.go:356] "Generic (PLEG): container finished" podID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerID="bfdc6246ef9576635f40aa600abe74f364919f6ff842637934def3dce6d5612c" exitCode=0
Oct 30 12:59:07.215139 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:07.214988    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4" event={"ID":"ac623bd1-d3de-47f4-9e06-a2261a160df8","Type":"ContainerDied","Data":"bfdc6246ef9576635f40aa600abe74f364919f6ff842637934def3dce6d5612c"}
Oct 30 12:59:07.215482 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:07.215463    5893 scope.go:122] "RemoveContainer" containerID="bfdc6246ef9576635f40aa600abe74f364919f6ff842637934def3dce6d5612c"
Oct 30 12:59:07.217068 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:07.217035    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:07.217229 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:07.217145    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:07.217229 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:07.217194    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:08.231099 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:08.231064    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4" event={"ID":"ac623bd1-d3de-47f4-9e06-a2261a160df8","Type":"ContainerStarted","Data":"b36f78528c6901194e30801cf5f1aaa6e4c25fb3ce31f0d0f6c9a6bd7c148f3b"}
Oct 30 12:59:08.927630 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:08.927593    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:59:08.928969 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:08.928945    5893 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 12:59:08.929082 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:08.928989    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4" podUID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 12:59:09.927496 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:09.927465    5893 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 12:59:09.927907 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:09.927517    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4" podUID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 12:59:10.927341 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:10.927308    5893 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 12:59:10.927541 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:10.927377    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4" podUID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 12:59:10.974857 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:10.974823    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:59:11.198155 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:11.198127    5893 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:56048->10.134.0.44:15021: read: connection reset by peer" start-of-body=
Oct 30 12:59:11.198331 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:11.198203    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:56048->10.134.0.44:15021: read: connection reset by peer"
Oct 30 12:59:11.289725 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:11.289691    5893 generic.go:356] "Generic (PLEG): container finished" podID="3873ae03-39a5-4f72-924d-6eba73017d74" containerID="436ec0a82c043c498497aaca1b3c15a6985f5696e4c78cbb2e4b495c8a804fde" exitCode=0
Oct 30 12:59:11.289891 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:11.289772    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" event={"ID":"3873ae03-39a5-4f72-924d-6eba73017d74","Type":"ContainerDied","Data":"436ec0a82c043c498497aaca1b3c15a6985f5696e4c78cbb2e4b495c8a804fde"}
Oct 30 12:59:11.290201 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:11.290188    5893 scope.go:122] "RemoveContainer" containerID="436ec0a82c043c498497aaca1b3c15a6985f5696e4c78cbb2e4b495c8a804fde"
Oct 30 12:59:11.291700 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:11.291672    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:11.291797 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:11.291748    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:11.291797 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:11.291783    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:11.927902 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:11.927855    5893 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 12:59:11.928337 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:11.927914    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4" podUID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 12:59:12.300634 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:12.300602    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" event={"ID":"3873ae03-39a5-4f72-924d-6eba73017d74","Type":"ContainerStarted","Data":"ae43bfa8b69daac932a13a06c9abc9a1f522d00965c60ce4573c63ebd32d2d53"}
Oct 30 12:59:12.479296 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:12.478665    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:59:12.479296 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:12.478793    5893 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 12:59:12.479296 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:12.478832    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 12:59:12.927769 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:12.927715    5893 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Oct 30 12:59:12.928150 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:12.927784    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4" podUID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Oct 30 12:59:13.476715 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:13.476679    5893 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 12:59:13.476997 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:13.476969    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 12:59:13.932652 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:13.932570    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:59:13.933768 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:13.933746    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 12:59:14.476752 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:14.476711    5893 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 12:59:14.476928 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:14.476781    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 12:59:15.480295 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:15.479739    5893 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 12:59:15.480295 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:15.479797    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 12:59:16.481716 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:16.481678    5893 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 12:59:16.482851 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:16.482818    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 12:59:17.477974 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:17.477938    5893 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Oct 30 12:59:17.478334 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:17.478258    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Oct 30 12:59:18.480659 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:18.480632    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:59:18.481078 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:18.480890    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:59:18.481541 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:18.481522    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 12:59:18.671303 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:59:18.671231    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:59:21.418027 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.417992    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"]
Oct 30 12:59:21.424520 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.424489    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.452711 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.452684    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7a3603\"/\"default-dockercfg-vbp7s\""
Oct 30 12:59:21.453366 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.453327    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-7a3603\"/\"openshift-service-ca.crt\""
Oct 30 12:59:21.453491 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.453437    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-7a3603\"/\"simulator-w5-7a3603-kserve-self-signed-certs\""
Oct 30 12:59:21.453658 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.453636    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-7a3603\"/\"kube-root-ca.crt\""
Oct 30 12:59:21.454903 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.454879    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"]
Oct 30 12:59:21.509494 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.509462    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f04e62-3baf-4ce1-b72e-2f6c5973a758-tls-certs\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.509494 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.509497    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-tmp-dir\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.509713 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.509571    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-home\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.509713 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.509669    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-model-cache\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.509713 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.509692    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-dshm\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.509855 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.509829    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jqz5\" (UniqueName: \"kubernetes.io/projected/35f04e62-3baf-4ce1-b72e-2f6c5973a758-kube-api-access-7jqz5\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.611504 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.611462    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-tmp-dir\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.611700 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.611536    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-home\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.611700 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.611615    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-model-cache\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.611700 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.611642    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-dshm\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.611870 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.611714    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jqz5\" (UniqueName: \"kubernetes.io/projected/35f04e62-3baf-4ce1-b72e-2f6c5973a758-kube-api-access-7jqz5\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.611870 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.611793    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f04e62-3baf-4ce1-b72e-2f6c5973a758-tls-certs\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.611954 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.611913    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-tmp-dir\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.611990 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.611948    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-home\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.611990 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.611974    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-model-cache\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.614235 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.614212    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-dshm\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.614615 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.614593    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f04e62-3baf-4ce1-b72e-2f6c5973a758-tls-certs\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.637468 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.637439    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jqz5\" (UniqueName: \"kubernetes.io/projected/35f04e62-3baf-4ce1-b72e-2f6c5973a758-kube-api-access-7jqz5\") pod \"simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") " pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:21.746210 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:21.746178    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:22.064788 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:22.064697    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"]
Oct 30 12:59:22.070449 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:59:22.069774    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35f04e62_3baf_4ce1_b72e_2f6c5973a758.slice/crio-86c29045abac02118a24b45ebd8229e79f8c1844d0e3861b7d68ab07327a1bef WatchSource:0}: Error finding container 86c29045abac02118a24b45ebd8229e79f8c1844d0e3861b7d68ab07327a1bef: Status 404 returned error can't find the container with id 86c29045abac02118a24b45ebd8229e79f8c1844d0e3861b7d68ab07327a1bef
Oct 30 12:59:22.374524 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:22.374428    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" event={"ID":"35f04e62-3baf-4ce1-b72e-2f6c5973a758","Type":"ContainerStarted","Data":"f6228c1c9bd637a43df74782e22ab935b62198c152877425d167be1aef24feb8"}
Oct 30 12:59:22.374524 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:22.374476    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" event={"ID":"35f04e62-3baf-4ce1-b72e-2f6c5973a758","Type":"ContainerStarted","Data":"86c29045abac02118a24b45ebd8229e79f8c1844d0e3861b7d68ab07327a1bef"}
Oct 30 12:59:22.746739 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:22.746701    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:24.390473 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:24.390429    5893 generic.go:356] "Generic (PLEG): container finished" podID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerID="940dfc9a35b250df4ee1698effe2d55299527c217dcd71b8c60c271745b5d810" exitCode=0
Oct 30 12:59:24.391402 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:24.391380    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87" event={"ID":"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b","Type":"ContainerDied","Data":"940dfc9a35b250df4ee1698effe2d55299527c217dcd71b8c60c271745b5d810"}
Oct 30 12:59:24.391764 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:24.391746    5893 scope.go:122] "RemoveContainer" containerID="940dfc9a35b250df4ee1698effe2d55299527c217dcd71b8c60c271745b5d810"
Oct 30 12:59:24.393222 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:24.393190    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:24.393347 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:24.393257    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:24.393347 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:24.393319    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:24.454206 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:24.454139    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podStartSLOduration=3.454119461 podStartE2EDuration="3.454119461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:22.412183155 +0000 UTC m=+803.995612847" watchObservedRunningTime="2026-10-30 12:59:24.454119461 +0000 UTC m=+806.037549156"
Oct 30 12:59:25.398497 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:25.398460    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87" event={"ID":"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b","Type":"ContainerStarted","Data":"a7069ced5288ee2af49a1220047147e8bdf11dac8c64776ba6be236c10036453"}
Oct 30 12:59:25.812560 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:25.812525    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:59:25.814107 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:25.814080    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:59:25.814226 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:25.814140    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87" podUID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 12:59:26.813243 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:26.813209    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:59:26.813702 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:26.813261    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87" podUID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 12:59:26.980810 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:26.980773    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:59:27.813122 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:27.813079    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:59:27.813346 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:27.813144    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87" podUID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 12:59:28.812846 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:28.812806    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:59:28.813025 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:28.812874    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87" podUID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 12:59:29.812669 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:29.812632    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body=
Oct 30 12:59:29.813102 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:29.812687    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87" podUID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused"
Oct 30 12:59:31.010960 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:31.010924    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:59:31.013518 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:31.013493    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 12:59:31.746841 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:31.746804    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:31.759684 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:31.759658    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:32.451500 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:32.451467    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 12:59:37.364893 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.364861    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"]
Oct 30 12:59:37.371224 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.371201    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.397257 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.397231    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c2ce75\"/\"kube-root-ca.crt\""
Oct 30 12:59:37.397481 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.397307    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c2ce75\"/\"openshift-service-ca.crt\""
Oct 30 12:59:37.397481 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.397350    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c2ce75\"/\"simulator-w2-c2ce75-kserve-self-signed-certs\""
Oct 30 12:59:37.397678 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.397361    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c2ce75\"/\"default-dockercfg-mkmbm\""
Oct 30 12:59:37.399462 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.399437    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"]
Oct 30 12:59:37.507698 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.507662    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-dshm\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.507848 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.507722    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-tmp-dir\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.507848 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.507779    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sprgw\" (UniqueName: \"kubernetes.io/projected/04bee432-e371-4029-91c6-d8331e62422e-kube-api-access-sprgw\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.507848 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.507815    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-model-cache\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.507980 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.507909    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-home\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.508065 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.508050    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bee432-e371-4029-91c6-d8331e62422e-tls-certs\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.609561 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.609532    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-model-cache\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.609755 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.609578    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-home\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.609755 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.609664    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bee432-e371-4029-91c6-d8331e62422e-tls-certs\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.609755 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.609686    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-dshm\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.609755 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.609722    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-tmp-dir\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.609755 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.609742    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sprgw\" (UniqueName: \"kubernetes.io/projected/04bee432-e371-4029-91c6-d8331e62422e-kube-api-access-sprgw\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.610173 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.610106    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-home\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.610530 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.610507    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-tmp-dir\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.610661 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.610563    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-model-cache\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.612609 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.612564    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-dshm\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.620359 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.620238    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bee432-e371-4029-91c6-d8331e62422e-tls-certs\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.636120 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.635954    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sprgw\" (UniqueName: \"kubernetes.io/projected/04bee432-e371-4029-91c6-d8331e62422e-kube-api-access-sprgw\") pod \"simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") " pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:37.711399 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:37.711368    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:38.018512 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:38.018468    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"]
Oct 30 12:59:38.021960 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:59:38.021928    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04bee432_e371_4029_91c6_d8331e62422e.slice/crio-b5e2fcb729a685a31c3f5db0c62736ff700749f92520db965f0c94ff62e0e345 WatchSource:0}: Error finding container b5e2fcb729a685a31c3f5db0c62736ff700749f92520db965f0c94ff62e0e345: Status 404 returned error can't find the container with id b5e2fcb729a685a31c3f5db0c62736ff700749f92520db965f0c94ff62e0e345
Oct 30 12:59:38.477045 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:38.477009    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" event={"ID":"04bee432-e371-4029-91c6-d8331e62422e","Type":"ContainerStarted","Data":"7be1a6c7d20dfee28530bb682ec845b45a40fe01d44b76b8cb2e853ee11312b4"}
Oct 30 12:59:38.477045 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:38.477045    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" event={"ID":"04bee432-e371-4029-91c6-d8331e62422e","Type":"ContainerStarted","Data":"b5e2fcb729a685a31c3f5db0c62736ff700749f92520db965f0c94ff62e0e345"}
Oct 30 12:59:38.516620 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:38.516539    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podStartSLOduration=1.516527631 podStartE2EDuration="1.516527631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:38.51599415 +0000 UTC m=+820.099423844" watchObservedRunningTime="2026-10-30 12:59:38.516527631 +0000 UTC m=+820.099957325"
Oct 30 12:59:38.711742 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:38.711710    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:47.712349 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:47.712301    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:47.725222 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:47.725197    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:48.556954 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:48.556925    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 12:59:48.655644 ip-10-0-141-232 kubenswrapper[5893]: E1030 12:59:48.655572    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 12:59:55.004267 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.004232    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"]
Oct 30 12:59:55.008468 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.008444    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.021378 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.021357    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c90b9e-openshift-default-dockercfg-6pcbf\""
Oct 30 12:59:55.035486 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.035260    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"]
Oct 30 12:59:55.109162 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.109118    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5979a707-99de-4c43-beac-15c684c66fd0-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.109162 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.109165    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5979a707-99de-4c43-beac-15c684c66fd0-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.109456 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.109260    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.109456 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.109293    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.109456 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.109367    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.109632 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.109522    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.109632 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.109553    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8kjp\" (UniqueName: \"kubernetes.io/projected/5979a707-99de-4c43-beac-15c684c66fd0-kube-api-access-b8kjp\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.109632 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.109591    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.109802 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.109646    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5979a707-99de-4c43-beac-15c684c66fd0-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.210602 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.210561    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.210602 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.210617    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5979a707-99de-4c43-beac-15c684c66fd0-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.210874 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.210675    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5979a707-99de-4c43-beac-15c684c66fd0-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.210874 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.210730    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5979a707-99de-4c43-beac-15c684c66fd0-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.210874 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.210771    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.210874 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.210801    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.210874 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.210829    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.211174 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.210995    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.211174 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.211008    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.211174 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.211028    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8kjp\" (UniqueName: \"kubernetes.io/projected/5979a707-99de-4c43-beac-15c684c66fd0-kube-api-access-b8kjp\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.211174 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.211098    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.211411 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.211346    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.211555 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.211535    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.211900 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.211871    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5979a707-99de-4c43-beac-15c684c66fd0-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.213658 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.213632    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.213848 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.213832    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5979a707-99de-4c43-beac-15c684c66fd0-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.232398 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.232369    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5979a707-99de-4c43-beac-15c684c66fd0-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.232552 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.232429    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8kjp\" (UniqueName: \"kubernetes.io/projected/5979a707-99de-4c43-beac-15c684c66fd0-kube-api-access-b8kjp\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.322212 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.322131    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:55.663782 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.663691    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"]
Oct 30 12:59:55.668376 ip-10-0-141-232 kubenswrapper[5893]: W1030 12:59:55.668343    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5979a707_99de_4c43_beac_15c684c66fd0.slice/crio-856ac146eae30d5c87b36651af3b3cd69127ee476f8a351ed648baa7e78fb822 WatchSource:0}: Error finding container 856ac146eae30d5c87b36651af3b3cd69127ee476f8a351ed648baa7e78fb822: Status 404 returned error can't find the container with id 856ac146eae30d5c87b36651af3b3cd69127ee476f8a351ed648baa7e78fb822
Oct 30 12:59:55.671195 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.671154    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:55.671291 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.671249    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:55.671336 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:55.671317    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 12:59:56.600037 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:56.599997    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh" event={"ID":"5979a707-99de-4c43-beac-15c684c66fd0","Type":"ContainerStarted","Data":"dfd9241f780724acd661b9fb0fb487fd672349c59d04b92f2c561d0901dbe6ad"}
Oct 30 12:59:56.600572 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:56.600551    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh" event={"ID":"5979a707-99de-4c43-beac-15c684c66fd0","Type":"ContainerStarted","Data":"856ac146eae30d5c87b36651af3b3cd69127ee476f8a351ed648baa7e78fb822"}
Oct 30 12:59:56.658296 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:56.657935    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh" podStartSLOduration=2.657915241 podStartE2EDuration="2.657915241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 12:59:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 12:59:56.657115051 +0000 UTC m=+838.240544745" watchObservedRunningTime="2026-10-30 12:59:56.657915241 +0000 UTC m=+838.241344934"
Oct 30 12:59:57.322492 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:57.322458    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 12:59:57.324233 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:57.324207    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:59:57.324429 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:57.324258    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh" podUID="5979a707-99de-4c43-beac-15c684c66fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 12:59:58.322760 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:58.322722    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:59:58.323154 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:58.322784    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh" podUID="5979a707-99de-4c43-beac-15c684c66fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 12:59:59.323267 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:59.323231    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 12:59:59.323691 ip-10-0-141-232 kubenswrapper[5893]: I1030 12:59:59.323313    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh" podUID="5979a707-99de-4c43-beac-15c684c66fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 13:00:00.323189 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:00.323154    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 13:00:00.323392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:00.323203    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh" podUID="5979a707-99de-4c43-beac-15c684c66fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 13:00:01.322509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:01.322473    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 13:00:01.322694 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:01.322529    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh" podUID="5979a707-99de-4c43-beac-15c684c66fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 13:00:02.322651 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:02.322617    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 13:00:02.323054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:02.322667    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh" podUID="5979a707-99de-4c43-beac-15c684c66fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 13:00:03.328989 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:03.328144    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 13:00:03.329610 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:03.329375    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 13:00:03.330771 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:03.330718    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 13:00:18.646012 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:00:18.645972    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:00:24.370255 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.370218    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"]
Oct 30 13:00:24.374381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.374360    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.387360 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.387336    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-7a3603-openshift-default-dockercfg-dh98z\""
Oct 30 13:00:24.400874 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.400824    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"]
Oct 30 13:00:24.521981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.521952    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.522165 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.521991    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.522165 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.522028    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd32d4c-3366-434c-8917-862060b8eda4-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.522243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.522165    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd32d4c-3366-434c-8917-862060b8eda4-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.522243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.522202    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r9cj\" (UniqueName: \"kubernetes.io/projected/9bd32d4c-3366-434c-8917-862060b8eda4-kube-api-access-8r9cj\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.522243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.522236    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.522377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.522311    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd32d4c-3366-434c-8917-862060b8eda4-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.522439 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.522424    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.522482 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.522469    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.624199 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.624108    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd32d4c-3366-434c-8917-862060b8eda4-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.624199 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.624148    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r9cj\" (UniqueName: \"kubernetes.io/projected/9bd32d4c-3366-434c-8917-862060b8eda4-kube-api-access-8r9cj\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.624199 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.624197    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.624590 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.624217    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd32d4c-3366-434c-8917-862060b8eda4-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.624590 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.624376    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.624590 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.624447    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.624590 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.624523    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.624590 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.624564    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.624847 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.624629    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd32d4c-3366-434c-8917-862060b8eda4-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.624847 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.624780    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.624960 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.624859    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.625162 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.625137    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.625239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.625082    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.625395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.625375    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd32d4c-3366-434c-8917-862060b8eda4-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.626905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.626879    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.627285 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.627247    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd32d4c-3366-434c-8917-862060b8eda4-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.663321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.663288    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd32d4c-3366-434c-8917-862060b8eda4-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.668252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.668227    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r9cj\" (UniqueName: \"kubernetes.io/projected/9bd32d4c-3366-434c-8917-862060b8eda4-kube-api-access-8r9cj\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:24.688139 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:24.688110    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:25.017981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:25.017951    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"]
Oct 30 13:00:25.021339 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:00:25.021313    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd32d4c_3366_434c_8917_862060b8eda4.slice/crio-11451709d7176ebb47d54290aa2c1786b28af428f5e43ffcda78e041bfc863c2 WatchSource:0}: Error finding container 11451709d7176ebb47d54290aa2c1786b28af428f5e43ffcda78e041bfc863c2: Status 404 returned error can't find the container with id 11451709d7176ebb47d54290aa2c1786b28af428f5e43ffcda78e041bfc863c2
Oct 30 13:00:25.023664 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:25.023632    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:25.023749 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:25.023696    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:25.023749 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:25.023726    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:25.787505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:25.787474    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt" event={"ID":"9bd32d4c-3366-434c-8917-862060b8eda4","Type":"ContainerStarted","Data":"85718abfee2812b0dd3dc4d45c870f81e4dbecbae2c6e210d5e87baae29d2f72"}
Oct 30 13:00:25.788010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:25.787529    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt" event={"ID":"9bd32d4c-3366-434c-8917-862060b8eda4","Type":"ContainerStarted","Data":"11451709d7176ebb47d54290aa2c1786b28af428f5e43ffcda78e041bfc863c2"}
Oct 30 13:00:25.852090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:25.852043    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt" podStartSLOduration=1.852026886 podStartE2EDuration="1.852026886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:25.83945697 +0000 UTC m=+867.422886681" watchObservedRunningTime="2026-10-30 13:00:25.852026886 +0000 UTC m=+867.435456633"
Oct 30 13:00:26.688673 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:26.688637    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:26.690140 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:26.690119    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 13:00:26.690242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:26.690168    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt" podUID="9bd32d4c-3366-434c-8917-862060b8eda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 13:00:27.688701 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:27.688662    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 13:00:27.689098 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:27.688732    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt" podUID="9bd32d4c-3366-434c-8917-862060b8eda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 13:00:28.688722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:28.688687    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 13:00:28.689130 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:28.688748    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt" podUID="9bd32d4c-3366-434c-8917-862060b8eda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 13:00:29.689453 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:29.689413    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 13:00:29.689915 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:29.689475    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt" podUID="9bd32d4c-3366-434c-8917-862060b8eda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 13:00:30.696710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:30.696671    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 13:00:30.697334 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:30.697299    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt" podUID="9bd32d4c-3366-434c-8917-862060b8eda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 13:00:31.688708 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:31.688661    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 13:00:31.688915 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:31.688726    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt" podUID="9bd32d4c-3366-434c-8917-862060b8eda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 13:00:32.703607 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:32.703575    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:32.705617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:32.705594    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:32.714556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:32.714528    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:00:40.151198 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.151163    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"]
Oct 30 13:00:40.157797 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.157771    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.185128 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.185041    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"]
Oct 30 13:00:40.288198 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.288162    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.288415 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.288342    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.288475 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.288410    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.288475 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.288456    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwphn\" (UniqueName: \"kubernetes.io/projected/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-kube-api-access-pwphn\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.288560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.288478    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.288560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.288494    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.288560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.288515    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.288689 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.288676    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.288723 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.288698    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.389626 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.389587    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.389626 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.389632    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.389877 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.389654    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.389877 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.389707    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.389877 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.389734    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.389877 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.389827    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.390077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.389944    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.390077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.389984    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.390077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.390047    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwphn\" (UniqueName: \"kubernetes.io/projected/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-kube-api-access-pwphn\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.390219 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.390130    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.390494 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.390466    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.390856 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.390519    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.390980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.390600    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.390980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.390671    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.392682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.392640    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.394486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.394459    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.412846 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.412774    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwphn\" (UniqueName: \"kubernetes.io/projected/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-kube-api-access-pwphn\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.414395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.414343    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.477470 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.477437    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:40.815846 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.815403    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"]
Oct 30 13:00:40.825122 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:00:40.825088    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff6f8a1_dc39_41ba_b620_60e3cd5723da.slice/crio-569b161f3eea8e88c59a2571725802573b0012380d4b246cada37980c50a6363 WatchSource:0}: Error finding container 569b161f3eea8e88c59a2571725802573b0012380d4b246cada37980c50a6363: Status 404 returned error can't find the container with id 569b161f3eea8e88c59a2571725802573b0012380d4b246cada37980c50a6363
Oct 30 13:00:40.828530 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.828497    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:40.828640 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.828568    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:40.828640 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.828603    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:40.898335 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:40.898299    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw" event={"ID":"0ff6f8a1-dc39-41ba-b620-60e3cd5723da","Type":"ContainerStarted","Data":"569b161f3eea8e88c59a2571725802573b0012380d4b246cada37980c50a6363"}
Oct 30 13:00:41.920298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:41.918789    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw" event={"ID":"0ff6f8a1-dc39-41ba-b620-60e3cd5723da","Type":"ContainerStarted","Data":"65d8ddcdd509cee474e85ce4ac37505173c1308eae7e3f9c67587a9cdbe6aa4a"}
Oct 30 13:00:42.478297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:42.477646    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:42.482295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:42.479905    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 13:00:42.482295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:42.479967    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw" podUID="0ff6f8a1-dc39-41ba-b620-60e3cd5723da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 13:00:43.477982 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:43.477943    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 13:00:43.478449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:43.478078    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw" podUID="0ff6f8a1-dc39-41ba-b620-60e3cd5723da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 13:00:44.478687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:44.478647    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 13:00:44.479118 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:44.478717    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw" podUID="0ff6f8a1-dc39-41ba-b620-60e3cd5723da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 13:00:45.478798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:45.478759    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 13:00:45.479251 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:45.478824    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw" podUID="0ff6f8a1-dc39-41ba-b620-60e3cd5723da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 13:00:46.477968 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:46.477931    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 13:00:46.478289 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:46.478239    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw" podUID="0ff6f8a1-dc39-41ba-b620-60e3cd5723da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 13:00:47.478610 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:47.478573    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 13:00:47.479048 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:47.478639    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw" podUID="0ff6f8a1-dc39-41ba-b620-60e3cd5723da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 13:00:48.483791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:48.483759    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:48.484191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:48.484171    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:48.484985 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:48.484964    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:00:48.545861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:48.545809    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw" podStartSLOduration=8.54579375 podStartE2EDuration="8.54579375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:42.044939566 +0000 UTC m=+883.628369260" watchObservedRunningTime="2026-10-30 13:00:48.54579375 +0000 UTC m=+890.129223443"
Oct 30 13:00:48.647104 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:00:48.647066    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:00:54.799303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:54.798968    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"]
Oct 30 13:00:54.829968 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:54.829942    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:54.903017 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:54.841235    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"]
Oct 30 13:00:54.903017 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:54.856639    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-67f56e-openshift-default-dockercfg-vvpqd\""
Oct 30 13:00:54.977330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:54.970913    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:54.977330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:54.971032    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:54.977330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:54.971098    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:54.977330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:54.971125    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e200c2e3-c88e-4707-a139-c5b8dfab3941-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:54.978110 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:54.977902    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:54.978110 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:54.977986    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:54.978110 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:54.978028    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwdd4\" (UniqueName: \"kubernetes.io/projected/e200c2e3-c88e-4707-a139-c5b8dfab3941-kube-api-access-bwdd4\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:54.978110 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:54.978069    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:54.978848 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:54.978468    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.083123 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.083015    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.104071 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.104036    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e200c2e3-c88e-4707-a139-c5b8dfab3941-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.109300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.108165    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.109300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.108254    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.109300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.108316    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwdd4\" (UniqueName: \"kubernetes.io/projected/e200c2e3-c88e-4707-a139-c5b8dfab3941-kube-api-access-bwdd4\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.109300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.108357    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.109300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.108395    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.109300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.108432    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.109300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.108568    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.112909 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.112879    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.114993 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.114571    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.124926 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.118827    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.124926 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.121416    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.124926 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.122200    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e200c2e3-c88e-4707-a139-c5b8dfab3941-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.169116 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.169043    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.171962 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.171909    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwdd4\" (UniqueName: \"kubernetes.io/projected/e200c2e3-c88e-4707-a139-c5b8dfab3941-kube-api-access-bwdd4\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.214303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.212866    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.227301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.226613    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.268530 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.268217    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:55.700697 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.700616    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"]
Oct 30 13:00:55.705307 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:00:55.705261    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode200c2e3_c88e_4707_a139_c5b8dfab3941.slice/crio-6a8496c905999d3f2b12c1ffbffa6a58e9cb5259e910a0be5fae86e6bc23865c WatchSource:0}: Error finding container 6a8496c905999d3f2b12c1ffbffa6a58e9cb5259e910a0be5fae86e6bc23865c: Status 404 returned error can't find the container with id 6a8496c905999d3f2b12c1ffbffa6a58e9cb5259e910a0be5fae86e6bc23865c
Oct 30 13:00:55.708375 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.708338    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:55.708504 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.708437    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:55.708504 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:55.708489    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:00:56.155337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:56.154203    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" event={"ID":"e200c2e3-c88e-4707-a139-c5b8dfab3941","Type":"ContainerStarted","Data":"8fae83a6c49bbbacfb3e2d63963639842ed9819299fe06fa2f163581b6b87bd4"}
Oct 30 13:00:56.155337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:56.154287    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" event={"ID":"e200c2e3-c88e-4707-a139-c5b8dfab3941","Type":"ContainerStarted","Data":"6a8496c905999d3f2b12c1ffbffa6a58e9cb5259e910a0be5fae86e6bc23865c"}
Oct 30 13:00:56.201540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:56.201470    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" podStartSLOduration=2.201450638 podStartE2EDuration="2.201450638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:00:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:00:56.198756546 +0000 UTC m=+897.782186266" watchObservedRunningTime="2026-10-30 13:00:56.201450638 +0000 UTC m=+897.784880332"
Oct 30 13:00:56.269365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:56.269324    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:00:56.271587 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:56.271554    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 13:00:56.271764 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:56.271612    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" podUID="e200c2e3-c88e-4707-a139-c5b8dfab3941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 13:00:57.272698 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:57.272666    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 13:00:57.273199 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:57.272772    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" podUID="e200c2e3-c88e-4707-a139-c5b8dfab3941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 13:00:58.269818 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:58.269786    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 13:00:58.270016 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:58.269846    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" podUID="e200c2e3-c88e-4707-a139-c5b8dfab3941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 13:00:58.614609 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:58.614530    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:00:58.615047 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:58.614949    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:00:58.631933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:58.631907    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:00:58.632393 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:58.632373    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:00:59.269723 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:59.269689    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 13:00:59.269906 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:00:59.269740    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" podUID="e200c2e3-c88e-4707-a139-c5b8dfab3941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 13:01:00.273299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:00.273032    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 13:01:00.273299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:00.273115    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" podUID="e200c2e3-c88e-4707-a139-c5b8dfab3941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 13:01:01.270302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:01.269338    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 13:01:01.270302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:01.269407    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" podUID="e200c2e3-c88e-4707-a139-c5b8dfab3941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 13:01:02.269474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:02.269439    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 13:01:02.269914 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:02.269489    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" podUID="e200c2e3-c88e-4707-a139-c5b8dfab3941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 13:01:03.274233 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:03.274202    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:01:03.274662 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:03.274516    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:01:03.275352 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:03.275334    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:01:09.544355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.542248    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"]
Oct 30 13:01:09.598297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.594592    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.658393 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.656717    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"]
Oct 30 13:01:09.673353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.673106    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.673353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.673158    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.673353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.673188    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.673353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.673216    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.673353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.673257    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.673353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.673298    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.673805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.673500    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.673805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.673549    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdlc7\" (UniqueName: \"kubernetes.io/projected/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-kube-api-access-jdlc7\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.673805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.673593    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.778298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.776101    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.778298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.776156    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.778298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.776193    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.778298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.776223    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.778298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.776288    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.778298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.776315    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.778298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.776380    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.778298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.776418    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdlc7\" (UniqueName: \"kubernetes.io/projected/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-kube-api-access-jdlc7\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.778298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.776465    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.778298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.777024    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.778298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.777318    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.793314 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.790648    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.793314 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.791360    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.804304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.803689    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.927316 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.919998    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:09.956298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.931556    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:10.004299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.988052    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdlc7\" (UniqueName: \"kubernetes.io/projected/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-kube-api-access-jdlc7\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:10.004299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:09.992869    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:10.285296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:10.284538    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:10.548852 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:10.548245    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:01:10.980055 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:10.979230    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:01:10.980055 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:10.979373    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:01:10.980055 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:10.979422    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:01:10.986568 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:10.985529    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"]
Oct 30 13:01:11.335979 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:11.335931    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6" event={"ID":"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe","Type":"ContainerStarted","Data":"93f6d5fa3de04e7c7603df4d31071462b1f91d26918c17e8c6d3b7bcc2ffa646"}
Oct 30 13:01:11.336157 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:11.336042    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6" event={"ID":"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe","Type":"ContainerStarted","Data":"6fa5e13fbe9e4ef021366c4c49327189cd785c1e5d5d2b315ca3c04b4027e869"}
Oct 30 13:01:11.394926 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:11.394858    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6" podStartSLOduration=2.394837531 podStartE2EDuration="2.394837531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:11.383640123 +0000 UTC m=+912.967069817" watchObservedRunningTime="2026-10-30 13:01:11.394837531 +0000 UTC m=+912.978267221"
Oct 30 13:01:12.327942 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:12.327906    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:12.329211 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:12.329183    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 13:01:12.329359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:12.329232    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6" podUID="aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 13:01:13.328207 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:13.328174    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 13:01:13.328615 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:13.328231    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6" podUID="aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 13:01:14.328682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:14.328644    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 13:01:14.329101 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:14.328696    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6" podUID="aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 13:01:15.328462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:15.328428    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 13:01:15.328644 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:15.328485    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6" podUID="aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 13:01:16.328655 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:16.328620    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 13:01:16.329148 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:16.328673    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6" podUID="aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 13:01:17.328071 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:17.328028    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 13:01:17.328292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:17.328105    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6" podUID="aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 13:01:18.332491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:18.332459    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:18.332939 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:18.332854    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:18.333511 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:18.333495    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:01:18.645122 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:01:18.645036    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:01:24.379138 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.379102    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"]
Oct 30 13:01:24.383845 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.383819    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.401219 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.399310    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-dd90cb-openshift-default-dockercfg-9nwsm\""
Oct 30 13:01:24.410360 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.410330    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"]
Oct 30 13:01:24.423784 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.423748    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.424025 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.423937    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.424025 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.423992    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.424234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.424027    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.424234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.424072    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.424234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.424100    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6px2v\" (UniqueName: \"kubernetes.io/projected/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-kube-api-access-6px2v\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.424234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.424140    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.424922 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.424180    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.424922 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.424768    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.525972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.525921    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.526247 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.526031    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.526247 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.526110    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.526247 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.526148    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.526247 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.526188    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.526247 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.526238    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.526653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.526266    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6px2v\" (UniqueName: \"kubernetes.io/projected/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-kube-api-access-6px2v\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.526653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.526335    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.526653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.526368    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.526881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.526804    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.527061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.527039    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.527546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.527522    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.528614 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.528576    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.529106 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.529070    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.530627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.530583    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.548825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.548795    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.560118 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.560086    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.560323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.560253    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6px2v\" (UniqueName: \"kubernetes.io/projected/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-kube-api-access-6px2v\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:24.715377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:24.709651    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:25.203352 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:25.202422    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"]
Oct 30 13:01:25.209332 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:01:25.209119    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80dd06d_424b_4ab3_ba9d_deb0d6ac166d.slice/crio-9b13655e6fde45fe7ba35f3b5f4bf7b69d21b7fe454fb9de1a6dd46c5ec08483 WatchSource:0}: Error finding container 9b13655e6fde45fe7ba35f3b5f4bf7b69d21b7fe454fb9de1a6dd46c5ec08483: Status 404 returned error can't find the container with id 9b13655e6fde45fe7ba35f3b5f4bf7b69d21b7fe454fb9de1a6dd46c5ec08483
Oct 30 13:01:25.213503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:25.213137    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:01:25.213503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:25.213249    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:01:25.213503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:25.213327    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:01:25.669375 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:25.669338    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl" event={"ID":"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d","Type":"ContainerStarted","Data":"f49d65f2261c0788a432ee7f3db6a65e19113d9ec72099c90ebda221617f2988"}
Oct 30 13:01:25.669982 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:25.669942    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl" event={"ID":"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d","Type":"ContainerStarted","Data":"9b13655e6fde45fe7ba35f3b5f4bf7b69d21b7fe454fb9de1a6dd46c5ec08483"}
Oct 30 13:01:25.712388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:25.712359    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:26.734881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:26.733193    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 13:01:26.734881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:26.733256    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl" podUID="c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 13:01:27.710447 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:27.710413    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 13:01:27.710642 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:27.710473    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl" podUID="c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 13:01:28.710576 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:28.710537    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 13:01:28.710943 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:28.710585    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl" podUID="c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 13:01:29.710628 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:29.710593    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 13:01:29.711016 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:29.710643    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl" podUID="c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 13:01:30.711264 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:30.711230    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 13:01:30.711711 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:30.711317    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl" podUID="c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 13:01:31.711060 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:31.711024    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Oct 30 13:01:31.711297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:31.711071    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl" podUID="c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Oct 30 13:01:32.715795 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:32.715766    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:32.716426 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:32.716051    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:32.716808 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:32.716788    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:01:32.774857 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:32.774804    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl" podStartSLOduration=8.774788716 podStartE2EDuration="8.774788716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:25.731686201 +0000 UTC m=+927.315115895" watchObservedRunningTime="2026-10-30 13:01:32.774788716 +0000 UTC m=+934.358218408"
Oct 30 13:01:36.221971 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.221930    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"]
Oct 30 13:01:36.228058 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.228033    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.252960 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.252930    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c90b9e\"/\"openshift-service-ca.crt\""
Oct 30 13:01:36.253125 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.252968    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c90b9e\"/\"kube-root-ca.crt\""
Oct 30 13:01:36.253125 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.253008    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c90b9e\"/\"default-dockercfg-v2t7t\""
Oct 30 13:01:36.253125 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.253008    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c90b9e\"/\"distinct-w0-c90b9e-kserve-self-signed-certs\""
Oct 30 13:01:36.257148 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.257118    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"]
Oct 30 13:01:36.294866 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.294835    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-home\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.295000 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.294911    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-tmp-dir\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.295052 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.295029    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-dshm\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.295093 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.295074    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-model-cache\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.295159 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.295139    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr5wm\" (UniqueName: \"kubernetes.io/projected/e3feb48e-aee7-4c06-8b5e-972c9da140d9-kube-api-access-mr5wm\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.295246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.295235    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3feb48e-aee7-4c06-8b5e-972c9da140d9-tls-certs\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.396999 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.396961    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-tmp-dir\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.407800 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.407775    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-dshm\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.408467 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.408446    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-model-cache\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.414585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.414556    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr5wm\" (UniqueName: \"kubernetes.io/projected/e3feb48e-aee7-4c06-8b5e-972c9da140d9-kube-api-access-mr5wm\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.415261 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.415236    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3feb48e-aee7-4c06-8b5e-972c9da140d9-tls-certs\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.415456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.415422    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-home\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.415822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.415802    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-home\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.415944 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.407662    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-tmp-dir\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.421784 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.409050    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-model-cache\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.423158 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.423122    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-dshm\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.425385 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.425362    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3feb48e-aee7-4c06-8b5e-972c9da140d9-tls-certs\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.443906 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.443872    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr5wm\" (UniqueName: \"kubernetes.io/projected/e3feb48e-aee7-4c06-8b5e-972c9da140d9-kube-api-access-mr5wm\") pod \"distinct-w0-c90b9e-kserve-c9f8d65-qtvvq\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") " pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.542470 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.542385    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:36.853296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:36.853190    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"]
Oct 30 13:01:36.861544 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:01:36.861516    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3feb48e_aee7_4c06_8b5e_972c9da140d9.slice/crio-57fea9fdff0fd89b96670067115a78c05e4c1b1894eab242e6035b6c0b989f8f WatchSource:0}: Error finding container 57fea9fdff0fd89b96670067115a78c05e4c1b1894eab242e6035b6c0b989f8f: Status 404 returned error can't find the container with id 57fea9fdff0fd89b96670067115a78c05e4c1b1894eab242e6035b6c0b989f8f
Oct 30 13:01:37.751155 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:37.751121    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" event={"ID":"e3feb48e-aee7-4c06-8b5e-972c9da140d9","Type":"ContainerStarted","Data":"2ac465e80bf7588928f41033ae55c414ce4e9ad2779ae8eab50404096010de7b"}
Oct 30 13:01:37.751615 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:37.751167    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" event={"ID":"e3feb48e-aee7-4c06-8b5e-972c9da140d9","Type":"ContainerStarted","Data":"57fea9fdff0fd89b96670067115a78c05e4c1b1894eab242e6035b6c0b989f8f"}
Oct 30 13:01:37.790470 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:37.790415    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podStartSLOduration=1.7904001680000001 podStartE2EDuration="1.790400168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:37.789754514 +0000 UTC m=+939.373184229" watchObservedRunningTime="2026-10-30 13:01:37.790400168 +0000 UTC m=+939.373829861"
Oct 30 13:01:38.543259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:38.543230    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:46.543160 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:46.543129    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:46.561022 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:46.560993    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:46.848285 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:46.848181    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:01:47.906696 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:47.906661    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"]
Oct 30 13:01:47.911659 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:47.911629    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:47.929385 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:47.929353    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-67f56e\"/\"kube-root-ca.crt\""
Oct 30 13:01:47.932525 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:47.932500    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-67f56e\"/\"openshift-service-ca.crt\""
Oct 30 13:01:47.939919 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:47.939895    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-67f56e\"/\"distinct-2-w4-67f56e-kserve-self-signed-certs\""
Oct 30 13:01:47.940031 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:47.939961    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-67f56e\"/\"default-dockercfg-7ldk6\""
Oct 30 13:01:47.951162 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:47.951128    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"]
Oct 30 13:01:47.956033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:47.956007    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-tmp-dir\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:47.956169 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:47.956093    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-dshm\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:47.956420 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:47.956398    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-home\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:47.956495 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:47.956443    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/514b6aca-9ccc-4f7e-833b-1a5d32969e37-tls-certs\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:47.956549 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:47.956492    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-model-cache\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:47.956549 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:47.956530    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl7nr\" (UniqueName: \"kubernetes.io/projected/514b6aca-9ccc-4f7e-833b-1a5d32969e37-kube-api-access-nl7nr\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:48.058202 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.058168    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-home\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:48.058202 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.058207    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/514b6aca-9ccc-4f7e-833b-1a5d32969e37-tls-certs\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:48.058460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.058237    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-model-cache\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:48.058460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.058300    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl7nr\" (UniqueName: \"kubernetes.io/projected/514b6aca-9ccc-4f7e-833b-1a5d32969e37-kube-api-access-nl7nr\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:48.058460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.058363    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-tmp-dir\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:48.058460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.058417    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-dshm\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:48.058647 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.058614    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-home\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:48.058734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.058714    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-model-cache\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:48.058774 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.058749    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-tmp-dir\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:48.060844 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.060826    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-dshm\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:48.060917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.060877    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/514b6aca-9ccc-4f7e-833b-1a5d32969e37-tls-certs\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:48.089061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.089030    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl7nr\" (UniqueName: \"kubernetes.io/projected/514b6aca-9ccc-4f7e-833b-1a5d32969e37-kube-api-access-nl7nr\") pod \"distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") " pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:48.228505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.228472    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:48.534310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.534186    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"]
Oct 30 13:01:48.538025 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:01:48.537996    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod514b6aca_9ccc_4f7e_833b_1a5d32969e37.slice/crio-d2111039b2c09a88fc86dd4ecd858b834b84e361536c8e59033bf912ee2d649e WatchSource:0}: Error finding container d2111039b2c09a88fc86dd4ecd858b834b84e361536c8e59033bf912ee2d649e: Status 404 returned error can't find the container with id d2111039b2c09a88fc86dd4ecd858b834b84e361536c8e59033bf912ee2d649e
Oct 30 13:01:48.648437 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:01:48.648397    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:01:48.853688 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.853535    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" event={"ID":"514b6aca-9ccc-4f7e-833b-1a5d32969e37","Type":"ContainerStarted","Data":"3df6f58f7d7c5b76d2b1b7dd355ad246856cbdb0df1502b0b4492abcac7f0633"}
Oct 30 13:01:48.853969 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.853929    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" event={"ID":"514b6aca-9ccc-4f7e-833b-1a5d32969e37","Type":"ContainerStarted","Data":"d2111039b2c09a88fc86dd4ecd858b834b84e361536c8e59033bf912ee2d649e"}
Oct 30 13:01:48.893947 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:48.893894    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podStartSLOduration=1.893880325 podStartE2EDuration="1.893880325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:01:48.892234304 +0000 UTC m=+950.475663998" watchObservedRunningTime="2026-10-30 13:01:48.893880325 +0000 UTC m=+950.477310018"
Oct 30 13:01:49.229526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:49.229497    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:58.229641 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:58.229606    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:58.244326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:58.244301    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:01:58.936513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:01:58.936485    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:02:06.694118 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.694086    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"]
Oct 30 13:02:06.701080 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.701046    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.732171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.732144    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8b46c9\"/\"kube-root-ca.crt\""
Oct 30 13:02:06.734609 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.734585    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"]
Oct 30 13:02:06.734764 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.734682    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-8b46c9\"/\"openshift-service-ca.crt\""
Oct 30 13:02:06.734829 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.734779    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8b46c9\"/\"default-dockercfg-kqjkl\""
Oct 30 13:02:06.734885 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.734872    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-8b46c9\"/\"distinct-2-w1-8b46c9-kserve-self-signed-certs\""
Oct 30 13:02:06.864396 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.864360    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-tmp-dir\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.864595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.864489    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfjpk\" (UniqueName: \"kubernetes.io/projected/05284048-51aa-4885-b6a2-5e88d09f0a7a-kube-api-access-lfjpk\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.864595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.864562    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-dshm\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.864595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.864593    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-home\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.864731 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.864688    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-model-cache\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.864787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.864758    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05284048-51aa-4885-b6a2-5e88d09f0a7a-tls-certs\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.966051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.966019    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05284048-51aa-4885-b6a2-5e88d09f0a7a-tls-certs\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.966233 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.966126    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-tmp-dir\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.966233 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.966189    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfjpk\" (UniqueName: \"kubernetes.io/projected/05284048-51aa-4885-b6a2-5e88d09f0a7a-kube-api-access-lfjpk\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.966361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.966246    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-dshm\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.966361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.966294    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-home\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.966361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.966344    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-model-cache\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.966649 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.966617    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-tmp-dir\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.966796 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.966726    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-model-cache\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.966796 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.966779    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-home\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.969145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.969126    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-dshm\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:06.969542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:06.969520    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05284048-51aa-4885-b6a2-5e88d09f0a7a-tls-certs\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:07.003606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:07.003579    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfjpk\" (UniqueName: \"kubernetes.io/projected/05284048-51aa-4885-b6a2-5e88d09f0a7a-kube-api-access-lfjpk\") pod \"distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") " pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:07.020456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:07.020428    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:07.345233 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:07.345180    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"]
Oct 30 13:02:07.347893 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:02:07.347864    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05284048_51aa_4885_b6a2_5e88d09f0a7a.slice/crio-34c09e9b3f701d44069c3990f68bda0e86c6bf30351f388a8f62fe22e80e79d3 WatchSource:0}: Error finding container 34c09e9b3f701d44069c3990f68bda0e86c6bf30351f388a8f62fe22e80e79d3: Status 404 returned error can't find the container with id 34c09e9b3f701d44069c3990f68bda0e86c6bf30351f388a8f62fe22e80e79d3
Oct 30 13:02:07.983953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:07.983915    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" event={"ID":"05284048-51aa-4885-b6a2-5e88d09f0a7a","Type":"ContainerStarted","Data":"fd0922579974bf9828f6e990203883a0cc9a03a3cbb03ace79aaabb1d28d72e8"}
Oct 30 13:02:07.984427 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:07.983963    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" event={"ID":"05284048-51aa-4885-b6a2-5e88d09f0a7a","Type":"ContainerStarted","Data":"34c09e9b3f701d44069c3990f68bda0e86c6bf30351f388a8f62fe22e80e79d3"}
Oct 30 13:02:08.020527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:08.020499    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:08.032827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:08.032780    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podStartSLOduration=2.032768783 podStartE2EDuration="2.032768783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:08.032603495 +0000 UTC m=+969.616033188" watchObservedRunningTime="2026-10-30 13:02:08.032768783 +0000 UTC m=+969.616198513"
Oct 30 13:02:17.020896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:17.020854    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:17.039683 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:17.039658    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:17.058389 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:17.058359    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:02:18.645521 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:02:18.645490    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:02:25.152600 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.152567    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"]
Oct 30 13:02:25.156911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.156886    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.190149 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.190120    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"]
Oct 30 13:02:25.230090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.230051    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.230090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.230097    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.230390 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.230228    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.230390 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.230253    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsbp9\" (UniqueName: \"kubernetes.io/projected/a39cee14-750a-47d1-8b5c-d362c150a6d1-kube-api-access-wsbp9\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.230390 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.230311    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39cee14-750a-47d1-8b5c-d362c150a6d1-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.230390 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.230338    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.230390 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.230375    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.230727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.230428    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.230727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.230530    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.331727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.331690    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.331895 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.331740    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.331895 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.331810    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.331895 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.331847    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsbp9\" (UniqueName: \"kubernetes.io/projected/a39cee14-750a-47d1-8b5c-d362c150a6d1-kube-api-access-wsbp9\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.331895 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.331882    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39cee14-750a-47d1-8b5c-d362c150a6d1-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.332090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.331928    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.332090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.331978    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.332090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.332000    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.332249 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.332163    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.332249 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.332174    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.332403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.332347    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.332462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.332396    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.332665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.332641    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39cee14-750a-47d1-8b5c-d362c150a6d1-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.332874 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.332852    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.334252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.334226    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.334465 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.334446    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.359950 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.359914    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsbp9\" (UniqueName: \"kubernetes.io/projected/a39cee14-750a-47d1-8b5c-d362c150a6d1-kube-api-access-wsbp9\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.360104 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.359991    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.471661 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.471620    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:25.801520 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.801412    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"]
Oct 30 13:02:25.806247 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:02:25.806209    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda39cee14_750a_47d1_8b5c_d362c150a6d1.slice/crio-5ed3d21db12db8797ff92299ab30afa0759c2b10830d2bccc3ebe6641ea299ac WatchSource:0}: Error finding container 5ed3d21db12db8797ff92299ab30afa0759c2b10830d2bccc3ebe6641ea299ac: Status 404 returned error can't find the container with id 5ed3d21db12db8797ff92299ab30afa0759c2b10830d2bccc3ebe6641ea299ac
Oct 30 13:02:25.808753 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.808719    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:25.808853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.808784    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:25.808853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:25.808812    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:26.116252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:26.116095    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n" event={"ID":"a39cee14-750a-47d1-8b5c-d362c150a6d1","Type":"ContainerStarted","Data":"e56c7ffefc4fb93b75b2112f3a71f0604204fb7097d50c467780e1e79b0bfdf3"}
Oct 30 13:02:26.116252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:26.116159    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n" event={"ID":"a39cee14-750a-47d1-8b5c-d362c150a6d1","Type":"ContainerStarted","Data":"5ed3d21db12db8797ff92299ab30afa0759c2b10830d2bccc3ebe6641ea299ac"}
Oct 30 13:02:26.168546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:26.168465    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n" podStartSLOduration=1.168445769 podStartE2EDuration="1.168445769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:26.163418513 +0000 UTC m=+987.746848233" watchObservedRunningTime="2026-10-30 13:02:26.168445769 +0000 UTC m=+987.751875463"
Oct 30 13:02:26.473625 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:26.473464    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:26.477293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:26.475345    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 13:02:26.477293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:26.475422    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n" podUID="a39cee14-750a-47d1-8b5c-d362c150a6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 13:02:27.472498 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:27.472461    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 13:02:27.472990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:27.472527    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n" podUID="a39cee14-750a-47d1-8b5c-d362c150a6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 13:02:28.474296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:28.473481    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 13:02:28.474296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:28.473555    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n" podUID="a39cee14-750a-47d1-8b5c-d362c150a6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 13:02:29.477299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:29.476425    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 13:02:29.477299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:29.476523    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n" podUID="a39cee14-750a-47d1-8b5c-d362c150a6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 13:02:30.472872 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:30.472834    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 13:02:30.473151 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:30.473121    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n" podUID="a39cee14-750a-47d1-8b5c-d362c150a6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 13:02:31.473738 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:31.473701    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 13:02:31.474159 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:31.473780    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n" podUID="a39cee14-750a-47d1-8b5c-d362c150a6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 13:02:32.472360 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:32.472316    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 13:02:32.472582 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:32.472398    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n" podUID="a39cee14-750a-47d1-8b5c-d362c150a6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 13:02:33.445115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.445082    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"]
Oct 30 13:02:33.450832 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.450802    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.461537 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.461512    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"]
Oct 30 13:02:33.470140 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.470110    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-dd90cb\"/\"distinct-w6-dd90cb-kserve-self-signed-certs\""
Oct 30 13:02:33.470320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.470196    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-dd90cb\"/\"default-dockercfg-69qp9\""
Oct 30 13:02:33.474473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.474449    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-dd90cb\"/\"openshift-service-ca.crt\""
Oct 30 13:02:33.474602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.474552    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-dd90cb\"/\"kube-root-ca.crt\""
Oct 30 13:02:33.627634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.627598    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-tmp-dir\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.627977 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.627951    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdd7c3d-c630-4a09-9882-d731177b67af-tls-certs\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.628166 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.628150    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-model-cache\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.628513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.628309    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlnfs\" (UniqueName: \"kubernetes.io/projected/1cdd7c3d-c630-4a09-9882-d731177b67af-kube-api-access-qlnfs\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.628736 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.628717    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-home\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.629370 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.629002    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-dshm\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.740029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.739997    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlnfs\" (UniqueName: \"kubernetes.io/projected/1cdd7c3d-c630-4a09-9882-d731177b67af-kube-api-access-qlnfs\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.740218 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.740048    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-home\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.740218 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.740160    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-dshm\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.740339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.740214    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-tmp-dir\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.740339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.740243    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdd7c3d-c630-4a09-9882-d731177b67af-tls-certs\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.740437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.740340    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-model-cache\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.740597 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.740565    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-home\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.740948 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.740922    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-tmp-dir\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.741132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.741105    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-model-cache\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.742976 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.742953    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-dshm\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.744837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.744814    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdd7c3d-c630-4a09-9882-d731177b67af-tls-certs\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.763503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.763474    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlnfs\" (UniqueName: \"kubernetes.io/projected/1cdd7c3d-c630-4a09-9882-d731177b67af-kube-api-access-qlnfs\") pod \"distinct-w6-dd90cb-kserve-67598d94c5-z4wjn\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") " pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:33.770243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:33.770215    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:34.093227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:34.093143    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"]
Oct 30 13:02:34.215832 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:34.215797    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" event={"ID":"1cdd7c3d-c630-4a09-9882-d731177b67af","Type":"ContainerStarted","Data":"8d38d01ae62db91836571ed200fb388eae872651e08e966083126afe0d344087"}
Oct 30 13:02:34.513467 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:34.513416    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:02:34.513935 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:34.513474    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n" podUID="a39cee14-750a-47d1-8b5c-d362c150a6d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:34.840421 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:34.840328    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:34.840865 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:34.840840    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:34.841803 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:34.841784    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:02:35.225946 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:35.225907    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" event={"ID":"1cdd7c3d-c630-4a09-9882-d731177b67af","Type":"ContainerStarted","Data":"f12c61d5f5ac0059e12469cd6a084375846c399cca6e89f5756c1cee69a19a2f"}
Oct 30 13:02:35.269459 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:35.269409    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podStartSLOduration=2.269394939 podStartE2EDuration="2.269394939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:35.268448627 +0000 UTC m=+996.851878321" watchObservedRunningTime="2026-10-30 13:02:35.269394939 +0000 UTC m=+996.852824661"
Oct 30 13:02:35.770585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:35.770540    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:40.225896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.225821    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"]
Oct 30 13:02:40.230825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.230792    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.286934 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.278990    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"]
Oct 30 13:02:40.339252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.339220    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.339474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.339288    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.339474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.339329    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.339474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.339454    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rtlv\" (UniqueName: \"kubernetes.io/projected/534048b5-3bb5-4eab-9357-9a464373036b-kube-api-access-9rtlv\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.339593 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.339517    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534048b5-3bb5-4eab-9357-9a464373036b-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.339634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.339583    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534048b5-3bb5-4eab-9357-9a464373036b-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.339679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.339636    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.339679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.339651    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534048b5-3bb5-4eab-9357-9a464373036b-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.339765 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.339699    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.370650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.370617    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"]
Oct 30 13:02:40.374793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.374768    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.404445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.404381    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"]
Oct 30 13:02:40.440801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.440769    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.440801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.440812    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.441074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.440835    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.441074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.440938    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534048b5-3bb5-4eab-9357-9a464373036b-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.441074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.440981    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534048b5-3bb5-4eab-9357-9a464373036b-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.441074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441016    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.441074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441037    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.441074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441052    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534048b5-3bb5-4eab-9357-9a464373036b-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.441074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441075    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.441602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441119    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.441602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441147    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.441602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441170    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.441602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441193    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czh9q\" (UniqueName: \"kubernetes.io/projected/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-kube-api-access-czh9q\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.441602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441253    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.441602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441303    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.441602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441332    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.441602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441369    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.441602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441402    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rtlv\" (UniqueName: \"kubernetes.io/projected/534048b5-3bb5-4eab-9357-9a464373036b-kube-api-access-9rtlv\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.441602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441551    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.442132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.441975    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.442263 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.442213    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.442491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.442449    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534048b5-3bb5-4eab-9357-9a464373036b-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.442738 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.442711    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.444359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.444338    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.444914 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.444889    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534048b5-3bb5-4eab-9357-9a464373036b-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.464496 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.464472    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534048b5-3bb5-4eab-9357-9a464373036b-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.464765 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.464744    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rtlv\" (UniqueName: \"kubernetes.io/projected/534048b5-3bb5-4eab-9357-9a464373036b-kube-api-access-9rtlv\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.543097 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.543005    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.543323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.543106    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.543323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.543143    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.543323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.543167    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.543323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.543228    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.543623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.543334    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.543623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.543377    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.543623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.543401    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czh9q\" (UniqueName: \"kubernetes.io/projected/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-kube-api-access-czh9q\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.543623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.543439    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.543623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.543557    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.544174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.544148    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.544340 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.544318    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.544417 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.544322    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.544513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.544494    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.546082 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.546049    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.546890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.546868    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.561183 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.561162    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:40.565812 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.565789    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czh9q\" (UniqueName: \"kubernetes.io/projected/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-kube-api-access-czh9q\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.566010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.565987    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.691253 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.691207    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:40.893341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.893240    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"]
Oct 30 13:02:40.897681 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:02:40.897655    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534048b5_3bb5_4eab_9357_9a464373036b.slice/crio-fe256e901fd28e351f0bee2ff02399f1d28997ba1ab5f6038030dfe0dc9ffe1e WatchSource:0}: Error finding container fe256e901fd28e351f0bee2ff02399f1d28997ba1ab5f6038030dfe0dc9ffe1e: Status 404 returned error can't find the container with id fe256e901fd28e351f0bee2ff02399f1d28997ba1ab5f6038030dfe0dc9ffe1e
Oct 30 13:02:40.900104 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.900075    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:40.900188 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.900140    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:40.900188 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:40.900169    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:41.021966 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:41.021937    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"]
Oct 30 13:02:41.024167 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:02:41.024134    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf82e4b8_5b4c_4505_a2f4_ee22f28cade0.slice/crio-8b4978f9a7cfabde3809d3d1e9523ecb597f7d7a1cb5b472838cd875df86feae WatchSource:0}: Error finding container 8b4978f9a7cfabde3809d3d1e9523ecb597f7d7a1cb5b472838cd875df86feae: Status 404 returned error can't find the container with id 8b4978f9a7cfabde3809d3d1e9523ecb597f7d7a1cb5b472838cd875df86feae
Oct 30 13:02:41.027362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:41.027319    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:41.027467 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:41.027414    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:41.027467 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:41.027461    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:41.307191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:41.307141    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64" event={"ID":"534048b5-3bb5-4eab-9357-9a464373036b","Type":"ContainerStarted","Data":"c5aecc44f4e5e95ec87e8fda8655e05a6dc5aa6ff034b5a786a93d0f283057d6"}
Oct 30 13:02:41.307693 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:41.307204    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64" event={"ID":"534048b5-3bb5-4eab-9357-9a464373036b","Type":"ContainerStarted","Data":"fe256e901fd28e351f0bee2ff02399f1d28997ba1ab5f6038030dfe0dc9ffe1e"}
Oct 30 13:02:41.309516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:41.309485    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m" event={"ID":"af82e4b8-5b4c-4505-a2f4-ee22f28cade0","Type":"ContainerStarted","Data":"ecd896b350da7c063d420cc845da81f82fdb0e1ed18d08a3e2ebbf5a8b8ad928"}
Oct 30 13:02:41.309674 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:41.309532    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m" event={"ID":"af82e4b8-5b4c-4505-a2f4-ee22f28cade0","Type":"ContainerStarted","Data":"8b4978f9a7cfabde3809d3d1e9523ecb597f7d7a1cb5b472838cd875df86feae"}
Oct 30 13:02:41.352267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:41.352213    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64" podStartSLOduration=1.3521996330000001 podStartE2EDuration="1.352199633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:41.348999777 +0000 UTC m=+1002.932429493" watchObservedRunningTime="2026-10-30 13:02:41.352199633 +0000 UTC m=+1002.935629325"
Oct 30 13:02:41.400527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:41.400476    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m" podStartSLOduration=1.400455411 podStartE2EDuration="1.400455411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:41.399664978 +0000 UTC m=+1002.983094707" watchObservedRunningTime="2026-10-30 13:02:41.400455411 +0000 UTC m=+1002.983885104"
Oct 30 13:02:41.564354 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:41.563522    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:41.576295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:41.566411    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 13:02:41.576295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:41.566471    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64" podUID="534048b5-3bb5-4eab-9357-9a464373036b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 13:02:41.692300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:41.691386    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:42.562608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:42.562574    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 13:02:42.599533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:42.599477    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64" podUID="534048b5-3bb5-4eab-9357-9a464373036b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 13:02:42.696626 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:42.696583    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 13:02:42.696828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:42.696663    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m" podUID="af82e4b8-5b4c-4505-a2f4-ee22f28cade0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 13:02:43.568127 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:43.568094    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 13:02:43.568764 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:43.568733    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64" podUID="534048b5-3bb5-4eab-9357-9a464373036b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 13:02:43.692763 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:43.692728    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 13:02:43.693045 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:43.693019    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m" podUID="af82e4b8-5b4c-4505-a2f4-ee22f28cade0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 13:02:43.775831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:43.775798    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:43.813653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:43.813626    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:44.409095 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:44.409065    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:44.565303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:44.564028    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 13:02:44.565303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:44.564088    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64" podUID="534048b5-3bb5-4eab-9357-9a464373036b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 13:02:44.693745 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:44.693654    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 13:02:44.694334 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:44.694299    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m" podUID="af82e4b8-5b4c-4505-a2f4-ee22f28cade0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 13:02:45.570300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:45.570168    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 13:02:45.570300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:45.570238    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64" podUID="534048b5-3bb5-4eab-9357-9a464373036b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 13:02:45.696142 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:45.696099    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 13:02:45.696859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:45.696830    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m" podUID="af82e4b8-5b4c-4505-a2f4-ee22f28cade0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 13:02:46.569756 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:46.569717    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 13:02:46.570060 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:46.570028    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64" podUID="534048b5-3bb5-4eab-9357-9a464373036b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 13:02:46.702355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:46.699710    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 13:02:46.702355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:46.699773    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m" podUID="af82e4b8-5b4c-4505-a2f4-ee22f28cade0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 13:02:47.561932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:47.561896    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 13:02:47.562230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:47.562197    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64" podUID="534048b5-3bb5-4eab-9357-9a464373036b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 13:02:47.692109 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:47.692073    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 13:02:47.692422 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:47.692383    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m" podUID="af82e4b8-5b4c-4505-a2f4-ee22f28cade0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 13:02:48.596589 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:48.596559    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:48.597947 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:48.597911    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:48.599520 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:48.599498    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:02:48.662768 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:02:48.662727    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:02:48.788717 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:48.788684    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:48.789147 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:48.789126    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:48.790142 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:48.790122    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:02:52.346257 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:52.346213    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:02:52.543956 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:52.541474    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:52.761308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:52.757368    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:52.856424 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:52.856371    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:52.929430 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:52.929380    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:53.049463 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:53.049361    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:53.470456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:53.469381    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:53.542442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:53.542401    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:53.543580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:53.543558    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:02:53.601301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:53.600404    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:53.711790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:53.711347    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:02:53.713222 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:53.712311    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:53.813936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:53.813856    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:02:53.935884 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:53.935640    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:02:54.678718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.678691    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"]
Oct 30 13:02:54.683416 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.683395    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.715952 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.715925    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"]
Oct 30 13:02:54.744099 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.744057    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d53f0f-efc7-43b4-86eb-0e105ea50355-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.744319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.744118    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzpqz\" (UniqueName: \"kubernetes.io/projected/54d53f0f-efc7-43b4-86eb-0e105ea50355-kube-api-access-pzpqz\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.744319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.744184    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.744319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.744236    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.744319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.744257    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.744573 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.744324    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.744573 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.744347    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.744573 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.744399    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.744573 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.744444    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.845748 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.845714    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d53f0f-efc7-43b4-86eb-0e105ea50355-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.845949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.845759    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzpqz\" (UniqueName: \"kubernetes.io/projected/54d53f0f-efc7-43b4-86eb-0e105ea50355-kube-api-access-pzpqz\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.845949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.845817    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.845949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.845860    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.845949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.845882    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.845949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.845919    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.845949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.845940    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.846265 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.845989    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.846265 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.846030    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.846575 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.846554    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.846637 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.846556    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d53f0f-efc7-43b4-86eb-0e105ea50355-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.846824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.846767    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.846971 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.846950    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.847203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.847176    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.848720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.848696    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.849602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.849563    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.869506 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.869441    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzpqz\" (UniqueName: \"kubernetes.io/projected/54d53f0f-efc7-43b4-86eb-0e105ea50355-kube-api-access-pzpqz\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.869806 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.869789    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:54.999411 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:54.999375    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:55.327869 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.327772    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"]
Oct 30 13:02:55.330806 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:02:55.330777    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d53f0f_efc7_43b4_86eb_0e105ea50355.slice/crio-6744914a1f71f39ef3d6308b2cb581d841b61cd1aa13e6e1d6e07baf33ffa905 WatchSource:0}: Error finding container 6744914a1f71f39ef3d6308b2cb581d841b61cd1aa13e6e1d6e07baf33ffa905: Status 404 returned error can't find the container with id 6744914a1f71f39ef3d6308b2cb581d841b61cd1aa13e6e1d6e07baf33ffa905
Oct 30 13:02:55.333491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.333460    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:55.333585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.333531    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:55.333585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.333566    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:55.521337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.521016    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7" event={"ID":"54d53f0f-efc7-43b4-86eb-0e105ea50355","Type":"ContainerStarted","Data":"258a98388a7b7ffb9881f24e22802c0efed2a9180adfd984bf84c469cd7ab482"}
Oct 30 13:02:55.521337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.521078    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7" event={"ID":"54d53f0f-efc7-43b4-86eb-0e105ea50355","Type":"ContainerStarted","Data":"6744914a1f71f39ef3d6308b2cb581d841b61cd1aa13e6e1d6e07baf33ffa905"}
Oct 30 13:02:55.589718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.589581    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7" podStartSLOduration=1.589559279 podStartE2EDuration="1.589559279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:55.570203572 +0000 UTC m=+1017.153633265" watchObservedRunningTime="2026-10-30 13:02:55.589559279 +0000 UTC m=+1017.172988973"
Oct 30 13:02:55.623407 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.623372    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"]
Oct 30 13:02:55.627968 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.627946    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.662333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.658458    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"]
Oct 30 13:02:55.662333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.658986    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.662333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.659185    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.662333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.659336    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.662333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.659406    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.662333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.659472    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.662333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.659503    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbp4d\" (UniqueName: \"kubernetes.io/projected/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-kube-api-access-pbp4d\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.662333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.659567    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.662333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.659627    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.662333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.659671    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.761414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.761379    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.761851 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.761430    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.761851 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.761462    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbp4d\" (UniqueName: \"kubernetes.io/projected/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-kube-api-access-pbp4d\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.761851 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.761534    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.761851 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.761619    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.761851 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.761678    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.761851 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.761734    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.762193 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.761855    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.762193 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.761919    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-data\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.762193 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.761943    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.762193 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.762117    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-workload-certs\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.762420 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.762190    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-workload-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.762476 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.762450    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-credential-socket\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.762862 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.762837    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istiod-ca-cert\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.764722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.764699    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-envoy\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.765090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.765068    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-podinfo\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.783405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.783379    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbp4d\" (UniqueName: \"kubernetes.io/projected/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-kube-api-access-pbp4d\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.783498 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.783401    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-token\") pod \"e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") " pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.943993 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.943921    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:55.999890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:55.999859    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:02:56.284068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:56.284038    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"]
Oct 30 13:02:56.286551 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:02:56.286513    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5757084c_bd02_4b5e_95ed_7ffe5ef2bd82.slice/crio-b004047c0a55fcd2ce1b2cc9038ed02a91f02498df195602d58dd1fa284d79eb WatchSource:0}: Error finding container b004047c0a55fcd2ce1b2cc9038ed02a91f02498df195602d58dd1fa284d79eb: Status 404 returned error can't find the container with id b004047c0a55fcd2ce1b2cc9038ed02a91f02498df195602d58dd1fa284d79eb
Oct 30 13:02:56.289562 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:56.289519    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:56.289657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:56.289615    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:56.289697 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:56.289657    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:02:56.533337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:56.533303    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4" event={"ID":"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82","Type":"ContainerStarted","Data":"1ba407d90a5e764a5a89a76e059e7fae2d5c65a2d9efc7ccb34e4920476cb5d2"}
Oct 30 13:02:56.533518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:56.533356    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4" event={"ID":"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82","Type":"ContainerStarted","Data":"b004047c0a55fcd2ce1b2cc9038ed02a91f02498df195602d58dd1fa284d79eb"}
Oct 30 13:02:56.588006 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:56.587941    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4" podStartSLOduration=1.587919205 podStartE2EDuration="1.587919205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:56.586728795 +0000 UTC m=+1018.170158505" watchObservedRunningTime="2026-10-30 13:02:56.587919205 +0000 UTC m=+1018.171348896"
Oct 30 13:02:56.948358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:56.947363    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:02:57.001369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:57.001334    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 13:02:57.001566 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:57.001396    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7" podUID="54d53f0f-efc7-43b4-86eb-0e105ea50355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 13:02:57.953298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:57.950307    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:02:57.953298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:57.950371    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4" podUID="5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 13:02:58.000683 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:58.000645    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 13:02:58.000851 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:58.000706    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7" podUID="54d53f0f-efc7-43b4-86eb-0e105ea50355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 13:02:58.948652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:58.948612    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:02:58.970437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:58.970393    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4" podUID="5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 13:02:58.999789 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:58.999762    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 13:02:59.000051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:59.000028    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7" podUID="54d53f0f-efc7-43b4-86eb-0e105ea50355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 13:02:59.951297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:59.948628    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:02:59.951297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:02:59.948697    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4" podUID="5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 13:03:00.002365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:00.000897    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 13:03:00.002365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:00.000953    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7" podUID="54d53f0f-efc7-43b4-86eb-0e105ea50355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 13:03:00.946916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:00.946876    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:03:00.947205 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:00.947172    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4" podUID="5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 13:03:01.008812 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:01.008780    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 13:03:01.009445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:01.009403    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7" podUID="54d53f0f-efc7-43b4-86eb-0e105ea50355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 13:03:01.944644 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:01.944611    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:03:01.944925 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:01.944898    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4" podUID="5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 13:03:02.000156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:02.000120    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 13:03:02.000357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:02.000181    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7" podUID="54d53f0f-efc7-43b4-86eb-0e105ea50355" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 13:03:02.945297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:02.944785    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:03:02.945297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:02.944847    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4" podUID="5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 13:03:03.084807 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:03.084775    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:03:03.086692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:03.086642    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:03:03.087690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:03.087669    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:03:04.183046 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:04.183017    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:03:04.184466 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:04.184441    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:03:04.185010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:04.184992    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:03:10.338362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.338107    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"]
Oct 30 13:03:10.403566 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.403531    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"]
Oct 30 13:03:10.403918 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.403898    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.584079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.583402    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.584079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.583461    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.584079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.583501    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.584079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.583566    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkbs2\" (UniqueName: \"kubernetes.io/projected/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-kube-api-access-fkbs2\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.584079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.583654    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.584079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.583723    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.584079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.583777    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.584079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.583816    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.584079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.583885    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.693325 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.693202    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkbs2\" (UniqueName: \"kubernetes.io/projected/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-kube-api-access-fkbs2\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.709347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.694676    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.709692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.709664    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.709883 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.709863    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.710023 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.710008    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.710393 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.710371    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.710636 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.710614    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.816379 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.816346    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.836402 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.836371    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.836774 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.712395    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-workload-certs\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.837011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.711893    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-workload-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.837149 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.835101    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istiod-ca-cert\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.837319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.712147    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-credential-socket\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:10.837750 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:10.837727    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-data\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:11.057319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:11.053258    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-podinfo\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:11.057575 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:11.055479    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-envoy\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:11.061334 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:11.056480    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-token\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:11.102298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:11.084191    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkbs2\" (UniqueName: \"kubernetes.io/projected/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-kube-api-access-fkbs2\") pod \"e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") " pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:11.234606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:11.234567    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:12.082193 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:12.080819    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:12.119313 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:12.117699    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:03:12.362294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:12.362186    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"]
Oct 30 13:03:12.439710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:12.438512    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:12.439710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:12.438609    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:12.439710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:12.438653    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:13.343295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:13.339495    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp" event={"ID":"42abda8f-3710-4edc-8bbb-f032aa9fa4f6","Type":"ContainerStarted","Data":"7da42dca14ce2b47c076bb91da94c1b5622306cf0a0072dc5047969d268ea1a7"}
Oct 30 13:03:13.343295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:13.339567    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp" event={"ID":"42abda8f-3710-4edc-8bbb-f032aa9fa4f6","Type":"ContainerStarted","Data":"52164e68337f71f5a7f8cf95586d8f55426f27df71505233b851e1a2beffec12"}
Oct 30 13:03:13.423993 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:13.421170    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp" podStartSLOduration=3.421152621 podStartE2EDuration="3.421152621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:13.42097975 +0000 UTC m=+1035.004409445" watchObservedRunningTime="2026-10-30 13:03:13.421152621 +0000 UTC m=+1035.004582317"
Oct 30 13:03:14.235506 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:14.235478    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:14.236808 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:14.236783    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:03:14.236920 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:14.236838    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp" podUID="42abda8f-3710-4edc-8bbb-f032aa9fa4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 13:03:15.235791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:15.235752    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:03:15.236367 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:15.235815    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp" podUID="42abda8f-3710-4edc-8bbb-f032aa9fa4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 13:03:16.235810 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:16.235774    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:03:16.236213 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:16.235850    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp" podUID="42abda8f-3710-4edc-8bbb-f032aa9fa4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 13:03:17.236252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:17.236208    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:03:17.236717 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:17.236303    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp" podUID="42abda8f-3710-4edc-8bbb-f032aa9fa4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 13:03:18.235504 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:18.235467    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:03:18.235688 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:18.235540    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp" podUID="42abda8f-3710-4edc-8bbb-f032aa9fa4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 13:03:18.673864 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:03:18.673775    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:03:19.235468 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:19.235437    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:03:19.235646 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:19.235487    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp" podUID="42abda8f-3710-4edc-8bbb-f032aa9fa4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 13:03:20.692249 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:20.692201    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:20.692969 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:20.692763    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:20.694027 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:20.694005    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:03:24.785642 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.785611    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"]
Oct 30 13:03:24.791049 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.791021    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.819459 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.819427    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"]
Oct 30 13:03:24.880489 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.880454    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.880680 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.880519    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.880680 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.880631    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.880822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.880694    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a0fd69-b57d-48ba-af24-229454270b24-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.880910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.880882    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lm4\" (UniqueName: \"kubernetes.io/projected/55a0fd69-b57d-48ba-af24-229454270b24-kube-api-access-n5lm4\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.881013 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.880996    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a0fd69-b57d-48ba-af24-229454270b24-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.881059 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.881030    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.881096 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.881062    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a0fd69-b57d-48ba-af24-229454270b24-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.881139 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.881096    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.982485 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.982457    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.982686 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.982535    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a0fd69-b57d-48ba-af24-229454270b24-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.982686 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.982597    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lm4\" (UniqueName: \"kubernetes.io/projected/55a0fd69-b57d-48ba-af24-229454270b24-kube-api-access-n5lm4\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.982827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.982696    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a0fd69-b57d-48ba-af24-229454270b24-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.982827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.982725    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.982827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.982755    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a0fd69-b57d-48ba-af24-229454270b24-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.982985 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.982910    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-workload-certs\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.982985 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.982914    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.983094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.983013    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.983094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.983066    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.983193 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.983104    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-credential-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.983193 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.983155    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-istio-data\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.983542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.983522    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-workload-socket\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.983919 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.983897    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a0fd69-b57d-48ba-af24-229454270b24-istiod-ca-cert\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.985478 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.985452    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-istio-envoy\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:24.985625 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:24.985606    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a0fd69-b57d-48ba-af24-229454270b24-istio-podinfo\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:25.012254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.012226    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a0fd69-b57d-48ba-af24-229454270b24-istio-token\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:25.012472 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.012456    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lm4\" (UniqueName: \"kubernetes.io/projected/55a0fd69-b57d-48ba-af24-229454270b24-kube-api-access-n5lm4\") pod \"e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") " pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:25.108137 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.108058    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:25.473512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.472779    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"]
Oct 30 13:03:25.473512 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:03:25.472960    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55a0fd69_b57d_48ba_af24_229454270b24.slice/crio-950dc1a5e4e5901d6905df4df96925c8c6b58c8e23db387d8d60332972bcb14f WatchSource:0}: Error finding container 950dc1a5e4e5901d6905df4df96925c8c6b58c8e23db387d8d60332972bcb14f: Status 404 returned error can't find the container with id 950dc1a5e4e5901d6905df4df96925c8c6b58c8e23db387d8d60332972bcb14f
Oct 30 13:03:25.486331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.484207    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:25.486331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.484328    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:25.486331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.484384    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:25.533335 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.531009    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"]
Oct 30 13:03:25.542331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.540620    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.597715 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.597688    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"]
Oct 30 13:03:25.718477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.717376    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.718477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.717427    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.718477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.717552    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.718477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.717628    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.718477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.717664    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln9l4\" (UniqueName: \"kubernetes.io/projected/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-kube-api-access-ln9l4\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.718477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.717706    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.718477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.717958    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.718477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.718077    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.718477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.718107    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.820291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.820235    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.820791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.820335    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.820791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.820460    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.820791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.820537    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.820791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.820573    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln9l4\" (UniqueName: \"kubernetes.io/projected/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-kube-api-access-ln9l4\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.820791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.820615    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.820791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.820653    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.820791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.820720    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.820791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.820738    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-workload-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.820791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.820751    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.821435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.821234    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-credential-socket\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.821435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.821249    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-workload-certs\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.821605 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.821583    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-data\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.822208 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.822185    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istiod-ca-cert\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.824447 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.824195    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-envoy\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.824447 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.824358    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-podinfo\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.842435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.842347    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln9l4\" (UniqueName: \"kubernetes.io/projected/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-kube-api-access-ln9l4\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.842684 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.842666    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-token\") pod \"e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") " pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:25.872465 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:25.872433    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:26.337344 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:26.327602    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"]
Oct 30 13:03:26.344295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:26.341092    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:26.381260 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:26.349489    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:26.381260 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:26.349558    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:03:26.633161 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:26.633118    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb" event={"ID":"4b23c761-9c6f-487d-b7dd-4529d70cf6fe","Type":"ContainerStarted","Data":"9604e3b5816c775ba0c3b71aa2207b0ff94c80eaa0e3cf60f0ee3dfdc5e57a74"}
Oct 30 13:03:26.668095 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:26.668051    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg" event={"ID":"55a0fd69-b57d-48ba-af24-229454270b24","Type":"ContainerStarted","Data":"f2969764e1aa078a02dbe30119fac0d97c314a879791fe6ebb09a26e1bf35a9d"}
Oct 30 13:03:26.668399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:26.668375    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg" event={"ID":"55a0fd69-b57d-48ba-af24-229454270b24","Type":"ContainerStarted","Data":"950dc1a5e4e5901d6905df4df96925c8c6b58c8e23db387d8d60332972bcb14f"}
Oct 30 13:03:26.827300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:26.818332    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg" podStartSLOduration=2.818309432 podStartE2EDuration="2.818309432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:26.736888965 +0000 UTC m=+1048.320318675" watchObservedRunningTime="2026-10-30 13:03:26.818309432 +0000 UTC m=+1048.401739127"
Oct 30 13:03:27.124032 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:27.123997    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:27.124425 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:27.124401    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 13:03:27.124607 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:27.124581    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg" podUID="55a0fd69-b57d-48ba-af24-229454270b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 13:03:27.726300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:27.720835    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb" event={"ID":"4b23c761-9c6f-487d-b7dd-4529d70cf6fe","Type":"ContainerStarted","Data":"ebb5a2c84e2488898ac8d9123cf4b24bd028ab9d9b9b20ce52aa15e5725a97cf"}
Oct 30 13:03:27.818298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:27.811056    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb" podStartSLOduration=2.811036085 podStartE2EDuration="2.811036085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:27.809796431 +0000 UTC m=+1049.393226125" watchObservedRunningTime="2026-10-30 13:03:27.811036085 +0000 UTC m=+1049.394465779"
Oct 30 13:03:27.881248 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:27.878635    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:27.881771 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:27.881416    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:03:27.881771 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:27.881595    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb" podUID="4b23c761-9c6f-487d-b7dd-4529d70cf6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 13:03:28.110359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:28.109699    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 13:03:28.110359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:28.109760    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg" podUID="55a0fd69-b57d-48ba-af24-229454270b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 13:03:28.911508 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:28.911473    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:03:28.911999 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:28.911534    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb" podUID="4b23c761-9c6f-487d-b7dd-4529d70cf6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 13:03:29.109055 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:29.109013    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 13:03:29.109308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:29.109068    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg" podUID="55a0fd69-b57d-48ba-af24-229454270b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 13:03:29.874298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:29.873705    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:03:29.874298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:29.873768    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb" podUID="4b23c761-9c6f-487d-b7dd-4529d70cf6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 13:03:30.111300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:30.110715    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 13:03:30.111300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:30.110776    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg" podUID="55a0fd69-b57d-48ba-af24-229454270b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 13:03:30.883722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:30.883682    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:03:30.883951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:30.883747    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb" podUID="4b23c761-9c6f-487d-b7dd-4529d70cf6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 13:03:31.108786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:31.108750    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 13:03:31.108954 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:31.108802    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg" podUID="55a0fd69-b57d-48ba-af24-229454270b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 13:03:31.883298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:31.873695    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:03:31.883298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:31.873758    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb" podUID="4b23c761-9c6f-487d-b7dd-4529d70cf6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 13:03:32.114990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:32.114855    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 13:03:32.114990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:32.114917    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg" podUID="55a0fd69-b57d-48ba-af24-229454270b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 13:03:32.874923 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:32.874654    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:03:32.874923 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:32.874718    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb" podUID="4b23c761-9c6f-487d-b7dd-4529d70cf6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 13:03:33.109080 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:33.109046    5893 patch_prober.go:29] interesting pod/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 13:03:33.109602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:33.109104    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg" podUID="55a0fd69-b57d-48ba-af24-229454270b24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 13:03:33.877735 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:33.877696    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:03:33.878020 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:33.877992    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb" podUID="4b23c761-9c6f-487d-b7dd-4529d70cf6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Oct 30 13:03:34.113761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:34.113722    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:34.114758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:34.114734    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:34.115642 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:34.115623    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:03:35.089686 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:35.089652    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:35.090085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:35.090061    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:35.091621 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:35.091601    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:03:48.702989 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:03:48.702950    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:03:49.831450 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:49.831403    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:49.944461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:49.944408    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:50.042530 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:50.041609    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:03:50.590433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:50.590381    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:50.835386 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:50.835347    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 13:03:50.836387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:50.836364    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:03:50.837894 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:50.837865    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:51.054951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:51.054900    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:51.742302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:51.162543    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:51.761098 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:51.162770    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:03:51.774300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:51.770729    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:03:51.778463 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:51.778433    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:03:51.804539 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:51.738405    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:51.849943 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:51.849908    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 13:03:52.103364 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:51.709657    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": context deadline exceeded"
Oct 30 13:03:52.104872 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:51.803051    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:03:52.105097 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:52.105067    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp" podUID="42abda8f-3710-4edc-8bbb-f032aa9fa4f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:52.105224 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:51.855319    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:52.116683 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:51.499621    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:52.117186 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:51.855388    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:52.124460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:52.124435    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:03:52.782296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:52.781355    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:52.782296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:52.781802    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:52.782296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:52.782074    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:52.816399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:52.812329    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:52.853511 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:52.853464    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 30 13:03:52.932301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:52.905436    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:03:53.141358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:53.116481    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 13:03:53.157980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:53.126596    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:53.160468 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:53.126634    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 13:03:53.180738 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:53.180711    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 13:03:53.782574 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:53.782530    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:53.804339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:53.786001    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:03:53.847358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:53.830365    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:53.871314 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:53.860556    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 30 13:03:54.142226 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:54.141690    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:54.179122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:54.179074    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 13:03:54.191301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:54.182173    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:03:54.796860 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:54.796815    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:03:54.806459 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:54.806360    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:54.850361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:54.840679    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:54.850361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:54.841052    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:54.870314 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:54.863913    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:55.112049 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:55.111950    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:03:55.113849 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:55.113830    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:03:55.142994 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:55.142954    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 13:03:55.183455 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:55.183415    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:55.207302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:55.203359    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:03:55.461348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:55.393513    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:03:55.662600 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:55.662567    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:03:55.665778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:55.665755    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:03:55.668554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:55.668535    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 13:03:55.671917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:03:55.670950    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 13:04:11.667557 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:11.667429    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:11.804421 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:11.804367    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:11.847414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:11.847362    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:11.848907 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:11.848876    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:11.874260 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:11.873364    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:12.161301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:12.160365    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 13:04:12.217259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:12.198171    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:12.217259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:12.198967    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:04:12.759301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:12.758165    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:12.823476 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:12.823422    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:04:12.824105 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:12.824078    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:04:12.858348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:12.856368    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:12.858348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:12.857633    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:04:12.858611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:12.858378    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:04:12.858611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:12.858442    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:04:12.886605 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:12.873545    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:13.007452 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:13.007419    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 13:04:13.211516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:13.211411    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:13.254302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:13.229302    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 13:04:13.267339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:13.253396    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 13:04:13.267665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:13.267645    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:04:13.333299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:13.253847    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:13.350774 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:13.350743    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:04:13.832299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:13.831405    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:04:13.858405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:13.858360    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:13.859051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:13.859019    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:13.974441 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:13.970933    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 30 13:04:14.246298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:14.243039    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 13:04:14.275299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:14.266410    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:14.358301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:14.357962    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:14.800414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:14.800370    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:14.863300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:14.862699    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:14.863795 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:14.863424    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:14.872000 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:14.871681    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:14.878515 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:14.878489    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 13:04:14.914065 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:14.913756    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 13:04:15.148356 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:15.144918    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:04:15.209689 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:15.209658    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:04:15.539866 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:15.538977    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:04:15.553018 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:15.552982    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:15.553967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:15.553938    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:15.613862 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:15.612598    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:04:15.636193 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:15.635203    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:04:18.647976 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:04:18.647935    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:04:27.886432 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:27.886385    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:27.923547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:27.922407    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:27.941482 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:27.941438    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:31.440168 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.440136    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"]
Oct 30 13:04:31.458606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.458581    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.472119 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.472085    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 13:04:31.472309 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.472155    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"]
Oct 30 13:04:31.477802 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.477773    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 13:04:31.477949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.477928    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jj6x8\""
Oct 30 13:04:31.485224 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.485203    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 13:04:31.589216 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.589182    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.589438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.589333    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b8dfd6-499a-4167-9718-cd5fa7a5c055-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.589438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.589380    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.589553 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.589467    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.589553 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.589523    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.589658 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.589621    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.589714 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.589655    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr2t6\" (UniqueName: \"kubernetes.io/projected/82b8dfd6-499a-4167-9718-cd5fa7a5c055-kube-api-access-xr2t6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.690874 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.690799    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b8dfd6-499a-4167-9718-cd5fa7a5c055-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.690874 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.690862    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.691085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.690901    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.691085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.690929    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.691085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.691009    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.691085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.691055    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr2t6\" (UniqueName: \"kubernetes.io/projected/82b8dfd6-499a-4167-9718-cd5fa7a5c055-kube-api-access-xr2t6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.691349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.691150    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.691902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.691791    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.693521 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.693500    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b8dfd6-499a-4167-9718-cd5fa7a5c055-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.693624 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.693576    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.693987 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.693973    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.694077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.694057    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.713625 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.713602    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.713987 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.713969    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr2t6\" (UniqueName: \"kubernetes.io/projected/82b8dfd6-499a-4167-9718-cd5fa7a5c055-kube-api-access-xr2t6\") pod \"istiod-openshift-gateway-7ddd9cfdc8-q8n9f\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:31.774598 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:31.774566    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:32.103578 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:32.103545    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"]
Oct 30 13:04:32.107624 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:04:32.107597    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b8dfd6_499a_4167_9718_cd5fa7a5c055.slice/crio-b43c14cb6f6bdad788f0ba3954d000d52aa64f78c7209f2e73cab5931d35ef20 WatchSource:0}: Error finding container b43c14cb6f6bdad788f0ba3954d000d52aa64f78c7209f2e73cab5931d35ef20: Status 404 returned error can't find the container with id b43c14cb6f6bdad788f0ba3954d000d52aa64f78c7209f2e73cab5931d35ef20
Oct 30 13:04:32.109717 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:32.109698    5893 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:04:32.794358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:32.794318    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f" event={"ID":"82b8dfd6-499a-4167-9718-cd5fa7a5c055","Type":"ContainerStarted","Data":"b43c14cb6f6bdad788f0ba3954d000d52aa64f78c7209f2e73cab5931d35ef20"}
Oct 30 13:04:34.707199 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:34.707167    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:34.707524 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:34.707229    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:04:34.813871 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:34.813838    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f" event={"ID":"82b8dfd6-499a-4167-9718-cd5fa7a5c055","Type":"ContainerStarted","Data":"7172f195e6ca62a7d78cf3913d1be62756ebb6508ac06036a4f238adc1f9ce97"}
Oct 30 13:04:34.814070 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:34.813927    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:34.880806 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:34.880756    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f" podStartSLOduration=1.284702999 podStartE2EDuration="3.880737766s" totalImagesPullingTime="2.596034767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:34.870125229 +0000 UTC m=+1116.453554934" watchObservedRunningTime="2026-10-30 13:04:34.880737766 +0000 UTC m=+1116.464167463"
Oct 30 13:04:35.837405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:35.837369    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:04:47.366426 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:47.366373    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:47.577429 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:47.577381    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:47.849317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:47.833083    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:47.868395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:47.868352    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:47.905443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:47.905394    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:47.937974 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:47.937932    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:48.044404 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:48.044357    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": context deadline exceeded"
Oct 30 13:04:48.379300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:48.374442    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:04:48.380222 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:48.380194    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:04:48.827093 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:48.827054    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:48.974438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:48.974308    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt" podUID="9bd32d4c-3366-434c-8917-862060b8eda4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:48.974728 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:48.926593    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:04:48.994008 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:48.926412    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:49.026428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:48.993933    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 30 13:04:49.315724 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:04:49.315673    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:04:49.607836 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:49.390618    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded"
Oct 30 13:04:49.641292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:49.471612    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 13:04:49.641292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:49.471794    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:04:49.641292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:49.491184    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:49.641292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:49.627808    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" podUID="e200c2e3-c88e-4707-a139-c5b8dfab3941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:49.641292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:49.549112    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:49.641292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:49.627936    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6" podUID="aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:49.685299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:49.660298    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:04:49.906416 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:49.840821    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:04:49.984462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:49.984393    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:04:49.984978 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:49.984946    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:50.027060 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:50.027002    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:04:50.027699 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:50.027652    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:50.028201 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:50.028162    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:50.028414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:50.028379    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m" podUID="af82e4b8-5b4c-4505-a2f4-ee22f28cade0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:50.388424 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:50.388366    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:04:50.415299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:50.390126    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64" podUID="534048b5-3bb5-4eab-9357-9a464373036b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:50.674348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:50.673403    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:50.738503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:50.714004    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:04:50.738503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:50.714061    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 13:04:50.764447 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:50.764420    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 13:04:50.872299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:50.860314    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:50.872299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:50.860650    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 13:04:50.872299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:50.861006    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:04:50.926751 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:50.926668    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:04:51.002608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:51.002570    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:51.003184 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:51.003160    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 13:04:51.036460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:51.036420    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:04:51.461562 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:51.461537    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 13:04:51.604864 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:51.604827    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:04:51.608311 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:04:51.608287    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:05:05.479300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:05.475406    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:05.643546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:05.643489    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:05.776730 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:05.776627    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:05.846396 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:05.846342    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:05.929579 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:05.929439    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:06.032629 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:06.031388    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:06.068303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:06.061397    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:06.479419 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:06.479021    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:06.481874 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:06.480350    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 13:05:06.743354 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:06.727488    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:06.825309 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:06.821597    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 30 13:05:06.847226 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:06.847178    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:06.966640 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:06.938863    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:05:07.148513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:07.137735    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 13:05:07.238876 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:07.238831    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:07.514305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:07.511891    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:07.594437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:07.594388    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Oct 30 13:05:07.613058 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:05:07.596430    5893 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.035s"
Oct 30 13:05:07.613946 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:07.613926    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:05:07.632917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:07.632791    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 13:05:07.634446 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:07.634417    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:05:07.658799 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:07.658765    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:05:07.659116 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:07.659098    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:05:07.659316 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:07.659302    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:05:07.767351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:07.759493    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:07.849300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:07.832317    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:07.954301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:07.868083    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded"
Oct 30 13:05:07.954871 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:07.954710    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:08.023469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:08.023369    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:08.146015 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:08.141022    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:08.268298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:08.254379    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:08.524705 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:08.524652    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:08.886414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:08.881188    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 30 13:05:08.923434 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:08.923386    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:08.959305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:08.956359    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:08.981929 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:08.981891    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:09.141777 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:09.141677    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:09.275875 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:09.275812    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:09.283652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:09.283616    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:05:09.606298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:09.603104    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 13:05:09.757859 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:05:09.725878    5893 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.202s"
Oct 30 13:05:09.757859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:09.725958    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:05:09.757859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:09.731354    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:09.969357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:09.926599    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:10.006929 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:09.978336    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:05:10.006929 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:09.983349    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:10.142387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:10.142343    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:10.314374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:10.306726    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:10.362340 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:10.329407    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:05:10.412294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:10.408554    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 13:05:10.420787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:10.420764    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:05:10.420965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:10.420918    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 13:05:10.912593 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:10.912564    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:05:11.333591 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:11.333543    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:11.608711 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:11.608616    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:05:18.646193 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:05:18.646151    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:05:25.474312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.474238    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"]
Oct 30 13:05:25.478549 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.478526    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.522555 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.522526    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"]
Oct 30 13:05:25.556294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.556240    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40bb5401-70e8-4806-8739-ae835600784e-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.556502 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.556343    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.556502 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.556376    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.556502 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.556487    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstlb\" (UniqueName: \"kubernetes.io/projected/40bb5401-70e8-4806-8739-ae835600784e-kube-api-access-rstlb\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.556657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.556560    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.556657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.556591    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.556727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.556668    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bb5401-70e8-4806-8739-ae835600784e-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.556777 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.556743    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.556829 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.556803    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bb5401-70e8-4806-8739-ae835600784e-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.658150 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.658109    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.658387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.658165    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.658387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.658227    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bb5401-70e8-4806-8739-ae835600784e-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.658387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.658311    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.658387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.658386    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bb5401-70e8-4806-8739-ae835600784e-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.658720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.658493    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40bb5401-70e8-4806-8739-ae835600784e-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.658720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.658556    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.658720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.658588    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.658720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.658641    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.658932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.658770    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstlb\" (UniqueName: \"kubernetes.io/projected/40bb5401-70e8-4806-8739-ae835600784e-kube-api-access-rstlb\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.659027 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.659005    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.659375 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.659353    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.659679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.659660    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.659977 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.659956    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bb5401-70e8-4806-8739-ae835600784e-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.661024 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.661004    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.661145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.661129    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bb5401-70e8-4806-8739-ae835600784e-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.686193 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.686161    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40bb5401-70e8-4806-8739-ae835600784e-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.686372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.686194    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstlb\" (UniqueName: \"kubernetes.io/projected/40bb5401-70e8-4806-8739-ae835600784e-kube-api-access-rstlb\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:25.797136 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:25.797049    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:26.136360 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:26.136333    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"]
Oct 30 13:05:26.138816 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:05:26.138786    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40bb5401_70e8_4806_8739_ae835600784e.slice/crio-a2fae7803b1d8ce0e1263223da7ed74d7065a7dc4f40535cf836c0db30f8f6fb WatchSource:0}: Error finding container a2fae7803b1d8ce0e1263223da7ed74d7065a7dc4f40535cf836c0db30f8f6fb: Status 404 returned error can't find the container with id a2fae7803b1d8ce0e1263223da7ed74d7065a7dc4f40535cf836c0db30f8f6fb
Oct 30 13:05:26.141791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:26.141749    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:05:26.141900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:26.141840    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:05:26.141900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:26.141882    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:05:27.019041 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:27.019005    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp" event={"ID":"40bb5401-70e8-4806-8739-ae835600784e","Type":"ContainerStarted","Data":"e7d936761dc30e6214efa57d8c9cc98ccd4341ab55966e195d51b743a59b516f"}
Oct 30 13:05:27.019615 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:27.019062    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp" event={"ID":"40bb5401-70e8-4806-8739-ae835600784e","Type":"ContainerStarted","Data":"a2fae7803b1d8ce0e1263223da7ed74d7065a7dc4f40535cf836c0db30f8f6fb"}
Oct 30 13:05:27.101721 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:27.101667    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp" podStartSLOduration=2.101653764 podStartE2EDuration="2.101653764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:27.090870705 +0000 UTC m=+1168.674300413" watchObservedRunningTime="2026-10-30 13:05:27.101653764 +0000 UTC m=+1168.685083457"
Oct 30 13:05:27.797578 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:27.797551    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:27.799581 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:27.799562    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 13:05:27.799735 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:27.799708    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp" podUID="40bb5401-70e8-4806-8739-ae835600784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 13:05:28.798255 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:28.798221    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 13:05:28.798681 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:28.798310    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp" podUID="40bb5401-70e8-4806-8739-ae835600784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 13:05:29.797809 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:29.797772    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 13:05:29.798113 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:29.797854    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp" podUID="40bb5401-70e8-4806-8739-ae835600784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 13:05:30.820795 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:30.820747    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 13:05:30.821477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:30.821426    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp" podUID="40bb5401-70e8-4806-8739-ae835600784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 13:05:31.798563 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:31.798525    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 13:05:31.798856 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:31.798827    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp" podUID="40bb5401-70e8-4806-8739-ae835600784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 13:05:32.195513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:32.195414    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:32.282409 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:32.282362    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:32.517295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:32.516353    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:32.894356 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:32.892347    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Oct 30 13:05:32.894356 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:32.893532    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 13:05:32.894356 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:32.893681    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp" podUID="40bb5401-70e8-4806-8739-ae835600784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 13:05:32.984525 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:32.984462    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:33.238297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:33.237975    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:33.269331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:33.239611    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:05:33.337299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:33.334586    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:05:33.373480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:33.357776    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:05:33.574391 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:33.556609    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:33.574391 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:33.559083    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 13:05:33.801344 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:33.801310    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 13:05:33.801628 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:33.801599    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp" podUID="40bb5401-70e8-4806-8739-ae835600784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 13:05:33.906707 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:33.906607    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:33.908743 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:33.908709    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:34.237299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.214334    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:05:34.237299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.215965    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:05:34.284094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.255432    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:34.304693 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.304643    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:34.350051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.350011    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:34.431404 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.431364    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5tgdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:34.450297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.431628    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" podUID="98d3dc24-762e-4470-a524-c5d7c822ea80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:34.450598 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.448450    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qg8wr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:34.450733 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.450706    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" podUID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:34.463305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.456408    5893 patch_prober.go:29] interesting pod/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:34.591392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.570583    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:34.868013 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.867924    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/health\": context deadline exceeded"
Oct 30 13:05:34.873315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.873258    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 13:05:34.873590 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.873553    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp" podUID="40bb5401-70e8-4806-8739-ae835600784e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 13:05:34.903307 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.899765    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" podUID="e200c2e3-c88e-4707-a139-c5b8dfab3941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:34.927793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.927655    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:34.931003 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.930969    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 30 13:05:34.997137 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.997103    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:05:34.997737 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:34.997717    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 13:05:35.224722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:35.224684    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:05:35.260913 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:35.260866    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:35.292975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:35.292941    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 13:05:35.356043 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:35.355378    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:35.375824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:35.375432    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8982g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:05:35.375824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:35.375504    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g" podUID="14fbc94f-88db-47d5-9cfe-d59e8e0c9874" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:35.501358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:35.494110    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:05:35.602987 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:35.602960    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 13:05:35.699764 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:35.699409    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:05:35.977389 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:35.972119    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:36.109354 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:36.100759    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:05:36.334355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:36.330429    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:05:36.470395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:36.470364    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:05:36.513763 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:36.513736    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:36.527333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:36.527252    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:36.773370 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:36.724628    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:05:40.544167 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.544078    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"]
Oct 30 13:05:40.548585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.548566    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.579222 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.579192    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"]
Oct 30 13:05:40.703571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.703533    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.703780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.703627    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r492t\" (UniqueName: \"kubernetes.io/projected/761ab5b8-a944-4b6a-acbc-327f8523ad5c-kube-api-access-r492t\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.703780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.703774    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.703900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.703826    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.703981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.703961    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.704110 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.704095    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.704217 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.704195    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.704318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.704242    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.704405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.704383    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.805980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.805897    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.805980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.805978    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.806196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.805996    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r492t\" (UniqueName: \"kubernetes.io/projected/761ab5b8-a944-4b6a-acbc-327f8523ad5c-kube-api-access-r492t\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.806196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.806049    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.806196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.806077    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.806196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.806130    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.806428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.806224    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.806428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.806336    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.806428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.806342    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-credential-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.806428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.806358    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-data\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.806428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.806379    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.806918 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.806899    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istiod-ca-cert\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.807137 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.807112    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-workload-certs\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.807229 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.807203    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-workload-socket\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.808963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.808930    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-podinfo\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.809066 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.808999    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-envoy\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.828861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.828836    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-token\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.830939 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.830916    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r492t\" (UniqueName: \"kubernetes.io/projected/761ab5b8-a944-4b6a-acbc-327f8523ad5c-kube-api-access-r492t\") pod \"e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") " pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:40.873787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:40.873762    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:41.216695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:41.216662    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"]
Oct 30 13:05:41.218486 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:05:41.218462    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761ab5b8_a944_4b6a_acbc_327f8523ad5c.slice/crio-02ef4f541fae79598030a4266fb34c0ecfd84f3039f0182c7f5b303858cbe05e WatchSource:0}: Error finding container 02ef4f541fae79598030a4266fb34c0ecfd84f3039f0182c7f5b303858cbe05e: Status 404 returned error can't find the container with id 02ef4f541fae79598030a4266fb34c0ecfd84f3039f0182c7f5b303858cbe05e
Oct 30 13:05:41.220874 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:41.220842    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:05:41.220989 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:41.220908    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:05:41.220989 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:41.220938    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:05:41.888911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:41.888877    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv" event={"ID":"761ab5b8-a944-4b6a-acbc-327f8523ad5c","Type":"ContainerStarted","Data":"ab35bb80c9f018540e860f7b0ad59fba29fe679e9cf8f718e12a5729e6b4e222"}
Oct 30 13:05:41.889381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:41.888934    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv" event={"ID":"761ab5b8-a944-4b6a-acbc-327f8523ad5c","Type":"ContainerStarted","Data":"02ef4f541fae79598030a4266fb34c0ecfd84f3039f0182c7f5b303858cbe05e"}
Oct 30 13:05:41.972968 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:41.972889    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv" podStartSLOduration=1.9728651080000001 podStartE2EDuration="1.972865108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:05:41.957761005 +0000 UTC m=+1183.541190699" watchObservedRunningTime="2026-10-30 13:05:41.972865108 +0000 UTC m=+1183.556294795"
Oct 30 13:05:42.874704 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:42.874662    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:42.876397 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:42.876364    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 13:05:42.876545 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:42.876436    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv" podUID="761ab5b8-a944-4b6a-acbc-327f8523ad5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 13:05:43.874474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:43.874440    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 13:05:43.874853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:43.874496    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv" podUID="761ab5b8-a944-4b6a-acbc-327f8523ad5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 13:05:44.875300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:44.874648    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 13:05:44.875300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:44.874707    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv" podUID="761ab5b8-a944-4b6a-acbc-327f8523ad5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 13:05:45.874811 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:45.874771    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 13:05:45.874964 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:45.874839    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv" podUID="761ab5b8-a944-4b6a-acbc-327f8523ad5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 13:05:46.875064 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:46.875028    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 13:05:46.875559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:46.875080    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv" podUID="761ab5b8-a944-4b6a-acbc-327f8523ad5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 13:05:47.874806 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:47.874767    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Oct 30 13:05:47.874988 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:47.874842    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv" podUID="761ab5b8-a944-4b6a-acbc-327f8523ad5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Oct 30 13:05:48.741002 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:05:48.740953    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:05:48.879840 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:48.879809    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:48.880215 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:48.880192    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:48.881050 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:48.881030    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:05:59.046731 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:58.860639    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:59.046731 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:58.861074    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Oct 30 13:05:59.046731 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:58.962514    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:59.112772 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:59.112727    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:59.244423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:59.244373    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:59.250307 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:59.246311    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:05:59.322353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:59.320056    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:59.518299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:59.514832    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:59.623287 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:59.623150    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": context deadline exceeded"
Oct 30 13:05:59.626221 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:59.626198    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:05:59.903603 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:59.903497    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:05:59.904529 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:59.903936    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:05:59.991300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:05:59.975833    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:06:00.035597 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.035565    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:06:00.121669 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.121628    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:06:00.122502 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.122479    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:06:00.259309 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.258490    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:06:00.301619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.269083    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:06:00.499297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.469261    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 13:06:00.499297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.476805    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 13:06:00.531659 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.516367    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:06:00.531659 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.517073    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:06:00.592600 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.570774    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:06:00.619331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.600507    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 13:06:00.619331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.600661    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:06:00.619331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.600712    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:06:00.629299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.626470    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:06:00.629299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.627332    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 13:06:00.775295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.749113    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:06:00.775727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.775707    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd"
Oct 30 13:06:00.775869 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.775858    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:06:00.776023 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.776000    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:06:00.787234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.770454    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:06:00.835052 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:06:00.835023    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:06:18.646611 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:06:18.646574    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:06:48.649468 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:06:48.649428    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:07:04.337897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:04.337402    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5tgdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:07:04.337897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:04.337487    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" podUID="98d3dc24-762e-4470-a524-c5d7c822ea80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:04.438749 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:04.438712    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qg8wr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:07:04.439085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:04.439052    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" podUID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:07:18.647407 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:07:18.647365    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:07:41.568919 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.568883    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"]
Oct 30 13:07:41.574356 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.574333    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"
Oct 30 13:07:41.605146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.605111    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1c0e6c42\""
Oct 30 13:07:41.611198 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.611163    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1c0e6c42-dockercfg-gq8xw\""
Oct 30 13:07:41.611408 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.611215    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"]
Oct 30 13:07:41.676351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.656741    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnfsv\" (UniqueName: \"kubernetes.io/projected/daa840f3-895e-492c-b87a-1369cf7f3504-kube-api-access-gnfsv\") pod \"payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc\" (UID: \"daa840f3-895e-492c-b87a-1369cf7f3504\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"
Oct 30 13:07:41.677428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.677376    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"]
Oct 30 13:07:41.695235 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.694898    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"
Oct 30 13:07:41.699918 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.699869    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"]
Oct 30 13:07:41.704420 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.704382    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa840f3-895e-492c-b87a-1369cf7f3504-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc\" (UID: \"daa840f3-895e-492c-b87a-1369cf7f3504\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"
Oct 30 13:07:41.850119 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.850020    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa840f3-895e-492c-b87a-1369cf7f3504-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc\" (UID: \"daa840f3-895e-492c-b87a-1369cf7f3504\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"
Oct 30 13:07:41.850363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.850135    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6edbec-078c-41cc-8df9-d9f3643a5fad-plugins-config-volume\") pod \"payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt\" (UID: \"7b6edbec-078c-41cc-8df9-d9f3643a5fad\") " pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"
Oct 30 13:07:41.850486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.850457    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zblws\" (UniqueName: \"kubernetes.io/projected/7b6edbec-078c-41cc-8df9-d9f3643a5fad-kube-api-access-zblws\") pod \"payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt\" (UID: \"7b6edbec-078c-41cc-8df9-d9f3643a5fad\") " pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"
Oct 30 13:07:41.850692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.850671    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnfsv\" (UniqueName: \"kubernetes.io/projected/daa840f3-895e-492c-b87a-1369cf7f3504-kube-api-access-gnfsv\") pod \"payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc\" (UID: \"daa840f3-895e-492c-b87a-1369cf7f3504\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"
Oct 30 13:07:41.851002 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.850967    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa840f3-895e-492c-b87a-1369cf7f3504-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc\" (UID: \"daa840f3-895e-492c-b87a-1369cf7f3504\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"
Oct 30 13:07:41.874083 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.874050    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnfsv\" (UniqueName: \"kubernetes.io/projected/daa840f3-895e-492c-b87a-1369cf7f3504-kube-api-access-gnfsv\") pod \"payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc\" (UID: \"daa840f3-895e-492c-b87a-1369cf7f3504\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"
Oct 30 13:07:41.888885 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.888858    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"
Oct 30 13:07:41.953447 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.952769    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6edbec-078c-41cc-8df9-d9f3643a5fad-plugins-config-volume\") pod \"payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt\" (UID: \"7b6edbec-078c-41cc-8df9-d9f3643a5fad\") " pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"
Oct 30 13:07:41.953447 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.953039    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zblws\" (UniqueName: \"kubernetes.io/projected/7b6edbec-078c-41cc-8df9-d9f3643a5fad-kube-api-access-zblws\") pod \"payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt\" (UID: \"7b6edbec-078c-41cc-8df9-d9f3643a5fad\") " pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"
Oct 30 13:07:41.955621 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.955556    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6edbec-078c-41cc-8df9-d9f3643a5fad-plugins-config-volume\") pod \"payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt\" (UID: \"7b6edbec-078c-41cc-8df9-d9f3643a5fad\") " pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"
Oct 30 13:07:41.984401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:41.984137    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zblws\" (UniqueName: \"kubernetes.io/projected/7b6edbec-078c-41cc-8df9-d9f3643a5fad-kube-api-access-zblws\") pod \"payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt\" (UID: \"7b6edbec-078c-41cc-8df9-d9f3643a5fad\") " pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"
Oct 30 13:07:42.059515 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:42.059482    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"
Oct 30 13:07:42.266144 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:42.265245    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"]
Oct 30 13:07:42.383838 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:42.383807    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"]
Oct 30 13:07:42.386171 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:07:42.386145    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b6edbec_078c_41cc_8df9_d9f3643a5fad.slice/crio-21b157b5c71df5de2026723a7fbbf396bc65c1b5bf5ee9a57ef6d4b8bed298ea WatchSource:0}: Error finding container 21b157b5c71df5de2026723a7fbbf396bc65c1b5bf5ee9a57ef6d4b8bed298ea: Status 404 returned error can't find the container with id 21b157b5c71df5de2026723a7fbbf396bc65c1b5bf5ee9a57ef6d4b8bed298ea
Oct 30 13:07:43.054545 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:43.054507    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt" event={"ID":"7b6edbec-078c-41cc-8df9-d9f3643a5fad","Type":"ContainerStarted","Data":"d675dd685f02772b525a61918b5e7ece7a674b700217cd20e7d900e2196a10cb"}
Oct 30 13:07:43.055120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:43.054573    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt" event={"ID":"7b6edbec-078c-41cc-8df9-d9f3643a5fad","Type":"ContainerStarted","Data":"21b157b5c71df5de2026723a7fbbf396bc65c1b5bf5ee9a57ef6d4b8bed298ea"}
Oct 30 13:07:43.055120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:43.054630    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"
Oct 30 13:07:43.058458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:43.058429    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc" event={"ID":"daa840f3-895e-492c-b87a-1369cf7f3504","Type":"ContainerStarted","Data":"aed17de4528e4482145b5857daa67c4f3154344e371b1e86b33ac32388aa14ae"}
Oct 30 13:07:43.058651 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:43.058637    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"
Oct 30 13:07:43.058769 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:43.058755    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc" event={"ID":"daa840f3-895e-492c-b87a-1369cf7f3504","Type":"ContainerStarted","Data":"35a476ae7ff99b85276bebeac0be8a3eb70a1687723fee98d6a6dc4d8ec2b53e"}
Oct 30 13:07:43.209213 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:43.209148    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt" podStartSLOduration=2.209130515 podStartE2EDuration="2.209130515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:43.207535948 +0000 UTC m=+1304.790965641" watchObservedRunningTime="2026-10-30 13:07:43.209130515 +0000 UTC m=+1304.792560210"
Oct 30 13:07:43.310708 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:43.310591    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc" podStartSLOduration=2.310573411 podStartE2EDuration="2.310573411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:43.307777966 +0000 UTC m=+1304.891207660" watchObservedRunningTime="2026-10-30 13:07:43.310573411 +0000 UTC m=+1304.894003107"
Oct 30 13:07:48.656692 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:07:48.656650    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:07:54.070094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:54.070061    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"
Oct 30 13:07:54.070543 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:07:54.070151    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"
Oct 30 13:08:08.820733 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:08.820698    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"]
Oct 30 13:08:08.845758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:08.845728    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:08.874705 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:08.870153    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"]
Oct 30 13:08:08.874705 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:08.871322    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-7a71ca-openshift-default-dockercfg-n9nrg\""
Oct 30 13:08:08.974610 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:08.974573    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-istio-data\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:08.974915 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:08.974886    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-credential-socket\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:08.975047 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:08.974948    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf058d4-f482-47b8-8610-6491dfd318c2-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:08.975427 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:08.975394    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-workload-certs\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:08.975550 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:08.975486    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrwnl\" (UniqueName: \"kubernetes.io/projected/abf058d4-f482-47b8-8610-6491dfd318c2-kube-api-access-rrwnl\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:08.975623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:08.975569    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf058d4-f482-47b8-8610-6491dfd318c2-istio-podinfo\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:08.975735 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:08.975716    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf058d4-f482-47b8-8610-6491dfd318c2-istio-token\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:08.975788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:08.975764    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-workload-socket\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:08.975891 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:08.975871    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-istio-envoy\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.077363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.077260    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf058d4-f482-47b8-8610-6491dfd318c2-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.077363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.077361    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-workload-certs\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.077572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.077422    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrwnl\" (UniqueName: \"kubernetes.io/projected/abf058d4-f482-47b8-8610-6491dfd318c2-kube-api-access-rrwnl\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.077572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.077480    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf058d4-f482-47b8-8610-6491dfd318c2-istio-podinfo\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.077681 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.077632    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf058d4-f482-47b8-8610-6491dfd318c2-istio-token\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.077739 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.077692    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-workload-socket\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.077797 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.077764    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-istio-envoy\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.077855 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.077824    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-workload-certs\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.077855 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.077836    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-istio-data\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.077972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.077908    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-credential-socket\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.078085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.078063    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-workload-socket\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.078306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.078260    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-istio-data\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.078306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.078296    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-credential-socket\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.078914 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.078888    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf058d4-f482-47b8-8610-6491dfd318c2-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.080664 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.080640    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-istio-envoy\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.080763 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.080683    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf058d4-f482-47b8-8610-6491dfd318c2-istio-podinfo\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.101646 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.101619    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrwnl\" (UniqueName: \"kubernetes.io/projected/abf058d4-f482-47b8-8610-6491dfd318c2-kube-api-access-rrwnl\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.101984 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.101960    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf058d4-f482-47b8-8610-6491dfd318c2-istio-token\") pod \"e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") " pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.217629 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.217600    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:09.571745 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.571716    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"]
Oct 30 13:08:09.576700 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:08:09.576669    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf058d4_f482_47b8_8610_6491dfd318c2.slice/crio-67a3ebc520c9e7e1b740c3a9a71f4e4b57622d0e154be9b0a7c3964ffa1d726d WatchSource:0}: Error finding container 67a3ebc520c9e7e1b740c3a9a71f4e4b57622d0e154be9b0a7c3964ffa1d726d: Status 404 returned error can't find the container with id 67a3ebc520c9e7e1b740c3a9a71f4e4b57622d0e154be9b0a7c3964ffa1d726d
Oct 30 13:08:09.580349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.580314    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:08:09.580435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.580385    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:08:09.580435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:09.580413    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:08:10.269023 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:10.268988    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85" event={"ID":"abf058d4-f482-47b8-8610-6491dfd318c2","Type":"ContainerStarted","Data":"eaadaa18fa1d64fd72ea8d05ffb70b378363f5e2dfe40038e30dce43fe16ca6a"}
Oct 30 13:08:10.269443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:10.269044    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85" event={"ID":"abf058d4-f482-47b8-8610-6491dfd318c2","Type":"ContainerStarted","Data":"67a3ebc520c9e7e1b740c3a9a71f4e4b57622d0e154be9b0a7c3964ffa1d726d"}
Oct 30 13:08:10.328363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:10.328314    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85" podStartSLOduration=2.328301156 podStartE2EDuration="2.328301156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:10.326036656 +0000 UTC m=+1331.909466349" watchObservedRunningTime="2026-10-30 13:08:10.328301156 +0000 UTC m=+1331.911730850"
Oct 30 13:08:11.218683 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:11.218652    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:11.224577 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:11.224549    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:11.275692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:11.275664    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:11.276847 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:11.276827    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:08:15.229993 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:15.229955    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"]
Oct 30 13:08:15.235863 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:15.235839    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"
Oct 30 13:08:15.264139 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:15.264105    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-7a71ca\""
Oct 30 13:08:15.264552 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:15.263980    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-7a71ca-dockercfg-q8x7l\""
Oct 30 13:08:15.274649 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:15.274614    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"]
Oct 30 13:08:15.291997 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:15.291495    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d8hx\" (UniqueName: \"kubernetes.io/projected/262cbe09-e912-4db9-b5ea-3d4ddbd073f5-kube-api-access-2d8hx\") pod \"payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch\" (UID: \"262cbe09-e912-4db9-b5ea-3d4ddbd073f5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"
Oct 30 13:08:15.291997 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:15.291669    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262cbe09-e912-4db9-b5ea-3d4ddbd073f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch\" (UID: \"262cbe09-e912-4db9-b5ea-3d4ddbd073f5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"
Oct 30 13:08:15.394324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:15.394024    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262cbe09-e912-4db9-b5ea-3d4ddbd073f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch\" (UID: \"262cbe09-e912-4db9-b5ea-3d4ddbd073f5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"
Oct 30 13:08:15.394523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:15.394335    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d8hx\" (UniqueName: \"kubernetes.io/projected/262cbe09-e912-4db9-b5ea-3d4ddbd073f5-kube-api-access-2d8hx\") pod \"payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch\" (UID: \"262cbe09-e912-4db9-b5ea-3d4ddbd073f5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"
Oct 30 13:08:15.400349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:15.395416    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262cbe09-e912-4db9-b5ea-3d4ddbd073f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch\" (UID: \"262cbe09-e912-4db9-b5ea-3d4ddbd073f5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"
Oct 30 13:08:15.436309 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:15.426510    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d8hx\" (UniqueName: \"kubernetes.io/projected/262cbe09-e912-4db9-b5ea-3d4ddbd073f5-kube-api-access-2d8hx\") pod \"payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch\" (UID: \"262cbe09-e912-4db9-b5ea-3d4ddbd073f5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"
Oct 30 13:08:15.551669 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:15.551573    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"
Oct 30 13:08:15.924560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:15.924491    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"]
Oct 30 13:08:15.925366 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:08:15.925339    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262cbe09_e912_4db9_b5ea_3d4ddbd073f5.slice/crio-4b70adfad3139c1c7f5c12fa64a8ebd7201586f363b7719decd01f7afe2b8e7e WatchSource:0}: Error finding container 4b70adfad3139c1c7f5c12fa64a8ebd7201586f363b7719decd01f7afe2b8e7e: Status 404 returned error can't find the container with id 4b70adfad3139c1c7f5c12fa64a8ebd7201586f363b7719decd01f7afe2b8e7e
Oct 30 13:08:16.310854 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:16.310736    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch" event={"ID":"262cbe09-e912-4db9-b5ea-3d4ddbd073f5","Type":"ContainerStarted","Data":"cd199e634d6ac0a0db6ea45adec319b323e7cb36cdde3ccea9969cee1429041a"}
Oct 30 13:08:16.310854 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:16.310798    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch" event={"ID":"262cbe09-e912-4db9-b5ea-3d4ddbd073f5","Type":"ContainerStarted","Data":"4b70adfad3139c1c7f5c12fa64a8ebd7201586f363b7719decd01f7afe2b8e7e"}
Oct 30 13:08:16.311335 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:16.310912    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"
Oct 30 13:08:16.351741 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:16.351674    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch" podStartSLOduration=1.35165378 podStartE2EDuration="1.35165378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:16.35046738 +0000 UTC m=+1337.933897092" watchObservedRunningTime="2026-10-30 13:08:16.35165378 +0000 UTC m=+1337.935083473"
Oct 30 13:08:18.650499 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:08:18.650466    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:08:20.588068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.588036    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"]
Oct 30 13:08:20.594941 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.594912    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.617529 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.617491    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-17fc433e-openshift-default-dockercfg-xc4pz\""
Oct 30 13:08:20.627654 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.627620    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-envoy\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.629863 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.629839    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-credential-socket\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.630033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.630018    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-data\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.630223 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.630199    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-workload-socket\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.630401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.630266    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-workload-certs\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.630401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.630358    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-podinfo\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.630516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.630408    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58cq6\" (UniqueName: \"kubernetes.io/projected/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-kube-api-access-58cq6\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.630516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.630449    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istiod-ca-cert\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.630626 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.630533    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-token\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.636121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.636093    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"]
Oct 30 13:08:20.733881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.733840    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-envoy\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.734094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.733919    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-credential-socket\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.734094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.733976    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-data\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.734094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.734063    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-workload-socket\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.734288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.734103    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-workload-certs\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.734288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.734152    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-podinfo\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.734288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.734201    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58cq6\" (UniqueName: \"kubernetes.io/projected/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-kube-api-access-58cq6\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.734288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.734242    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istiod-ca-cert\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.734514 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.734336    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-token\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.735368 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.735011    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-credential-socket\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.736742 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.736700    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istiod-ca-cert\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.737558 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.737516    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-data\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.737936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.737910    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-workload-socket\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.738158 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.738139    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-workload-certs\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.804292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.776935    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-podinfo\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.804292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.799012    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58cq6\" (UniqueName: \"kubernetes.io/projected/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-kube-api-access-58cq6\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.805089 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.805062    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-envoy\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.826637 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.815774    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-token\") pod \"e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") " pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:20.912538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:20.912264    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:21.283364 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:21.283339    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"]
Oct 30 13:08:21.285094 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:08:21.285067    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b1e4e2_ea57_4b68_b2bb_0a0de945c8a7.slice/crio-654fdac0024f4fc233d3f40ff360ff2c97685db1434f41fce140b5c8e17c3429 WatchSource:0}: Error finding container 654fdac0024f4fc233d3f40ff360ff2c97685db1434f41fce140b5c8e17c3429: Status 404 returned error can't find the container with id 654fdac0024f4fc233d3f40ff360ff2c97685db1434f41fce140b5c8e17c3429
Oct 30 13:08:21.287822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:21.287783    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:08:21.287953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:21.287867    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:08:21.287953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:21.287915    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:08:21.352197 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:21.352154    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7" event={"ID":"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7","Type":"ContainerStarted","Data":"654fdac0024f4fc233d3f40ff360ff2c97685db1434f41fce140b5c8e17c3429"}
Oct 30 13:08:22.360196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:22.360156    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7" event={"ID":"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7","Type":"ContainerStarted","Data":"0fc25ddcb9028983f0fa28caede55b2289399949b584b34089f7e4f75e2f1763"}
Oct 30 13:08:22.416614 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:22.416546    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7" podStartSLOduration=2.416527813 podStartE2EDuration="2.416527813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:22.415445997 +0000 UTC m=+1343.998875712" watchObservedRunningTime="2026-10-30 13:08:22.416527813 +0000 UTC m=+1343.999957508"
Oct 30 13:08:22.913372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:22.913340    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:22.921455 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:22.921424    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:23.367553 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:23.367515    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:23.368848 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:23.368821    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:08:27.101155 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:27.101115    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"]
Oct 30 13:08:27.105755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:27.105731    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"
Oct 30 13:08:27.127442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:27.127412    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"]
Oct 30 13:08:27.134026 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:27.133999    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-17fc433e-dockercfg-kjcrc\""
Oct 30 13:08:27.134026 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:27.133999    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-17fc433e\""
Oct 30 13:08:27.188936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:27.188747    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzfh8\" (UniqueName: \"kubernetes.io/projected/d1a54829-a667-49b5-9f16-f8ae8b10218e-kube-api-access-kzfh8\") pod \"payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c\" (UID: \"d1a54829-a667-49b5-9f16-f8ae8b10218e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"
Oct 30 13:08:27.189456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:27.189425    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a54829-a667-49b5-9f16-f8ae8b10218e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c\" (UID: \"d1a54829-a667-49b5-9f16-f8ae8b10218e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"
Oct 30 13:08:27.301346 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:27.297617    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzfh8\" (UniqueName: \"kubernetes.io/projected/d1a54829-a667-49b5-9f16-f8ae8b10218e-kube-api-access-kzfh8\") pod \"payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c\" (UID: \"d1a54829-a667-49b5-9f16-f8ae8b10218e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"
Oct 30 13:08:27.301346 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:27.297776    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a54829-a667-49b5-9f16-f8ae8b10218e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c\" (UID: \"d1a54829-a667-49b5-9f16-f8ae8b10218e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"
Oct 30 13:08:27.301346 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:27.300197    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a54829-a667-49b5-9f16-f8ae8b10218e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c\" (UID: \"d1a54829-a667-49b5-9f16-f8ae8b10218e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"
Oct 30 13:08:27.342697 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:27.336908    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzfh8\" (UniqueName: \"kubernetes.io/projected/d1a54829-a667-49b5-9f16-f8ae8b10218e-kube-api-access-kzfh8\") pod \"payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c\" (UID: \"d1a54829-a667-49b5-9f16-f8ae8b10218e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"
Oct 30 13:08:27.367474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:27.367391    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"
Oct 30 13:08:27.421415 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:27.420998    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"
Oct 30 13:08:27.990051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:27.990024    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"]
Oct 30 13:08:27.992032 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:08:27.992010    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a54829_a667_49b5_9f16_f8ae8b10218e.slice/crio-d298fbb07340a2b27467cf9e6c2820817084b489a7a311d0b23f11f8e14b211b WatchSource:0}: Error finding container d298fbb07340a2b27467cf9e6c2820817084b489a7a311d0b23f11f8e14b211b: Status 404 returned error can't find the container with id d298fbb07340a2b27467cf9e6c2820817084b489a7a311d0b23f11f8e14b211b
Oct 30 13:08:28.427174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:28.427081    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c" event={"ID":"d1a54829-a667-49b5-9f16-f8ae8b10218e","Type":"ContainerStarted","Data":"935d8159e2d174dc81c608273d50b5c8f368cb559d762195e55cfc12de02c27c"}
Oct 30 13:08:28.427174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:28.427142    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c" event={"ID":"d1a54829-a667-49b5-9f16-f8ae8b10218e","Type":"ContainerStarted","Data":"d298fbb07340a2b27467cf9e6c2820817084b489a7a311d0b23f11f8e14b211b"}
Oct 30 13:08:28.427174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:28.427170    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"
Oct 30 13:08:28.469416 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:28.469351    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c" podStartSLOduration=1.469332152 podStartE2EDuration="1.469332152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:28.465952238 +0000 UTC m=+1350.049381929" watchObservedRunningTime="2026-10-30 13:08:28.469332152 +0000 UTC m=+1350.052761845"
Oct 30 13:08:32.303332 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.303297    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"]
Oct 30 13:08:32.307843 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.307817    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"
Oct 30 13:08:32.324892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.324871    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-0b2c92\""
Oct 30 13:08:32.334451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.334397    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/855e2bc2-43d6-4b6a-827f-a778178d6e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6\" (UID: \"855e2bc2-43d6-4b6a-827f-a778178d6e23\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"
Oct 30 13:08:32.334627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.334562    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj6n7\" (UniqueName: \"kubernetes.io/projected/855e2bc2-43d6-4b6a-827f-a778178d6e23-kube-api-access-xj6n7\") pod \"payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6\" (UID: \"855e2bc2-43d6-4b6a-827f-a778178d6e23\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"
Oct 30 13:08:32.336693 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.336647    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-0b2c92-dockercfg-zlb9c\""
Oct 30 13:08:32.379310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.378900    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"]
Oct 30 13:08:32.397296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.385440    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"]
Oct 30 13:08:32.397296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.390889    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"
Oct 30 13:08:32.475353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.456499    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj6n7\" (UniqueName: \"kubernetes.io/projected/855e2bc2-43d6-4b6a-827f-a778178d6e23-kube-api-access-xj6n7\") pod \"payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6\" (UID: \"855e2bc2-43d6-4b6a-827f-a778178d6e23\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"
Oct 30 13:08:32.475353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.456764    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br2jr\" (UniqueName: \"kubernetes.io/projected/cccf12d4-0554-4ea8-95f3-ddf4e73317eb-kube-api-access-br2jr\") pod \"payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n\" (UID: \"cccf12d4-0554-4ea8-95f3-ddf4e73317eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"
Oct 30 13:08:32.475353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.456815    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cccf12d4-0554-4ea8-95f3-ddf4e73317eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n\" (UID: \"cccf12d4-0554-4ea8-95f3-ddf4e73317eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"
Oct 30 13:08:32.475353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.456889    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/855e2bc2-43d6-4b6a-827f-a778178d6e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6\" (UID: \"855e2bc2-43d6-4b6a-827f-a778178d6e23\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"
Oct 30 13:08:32.475353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.457964    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/855e2bc2-43d6-4b6a-827f-a778178d6e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6\" (UID: \"855e2bc2-43d6-4b6a-827f-a778178d6e23\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"
Oct 30 13:08:32.500301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.491332    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"]
Oct 30 13:08:32.573559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.573430    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br2jr\" (UniqueName: \"kubernetes.io/projected/cccf12d4-0554-4ea8-95f3-ddf4e73317eb-kube-api-access-br2jr\") pod \"payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n\" (UID: \"cccf12d4-0554-4ea8-95f3-ddf4e73317eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"
Oct 30 13:08:32.573938 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.573885    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cccf12d4-0554-4ea8-95f3-ddf4e73317eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n\" (UID: \"cccf12d4-0554-4ea8-95f3-ddf4e73317eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"
Oct 30 13:08:32.576552 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.575375    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cccf12d4-0554-4ea8-95f3-ddf4e73317eb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n\" (UID: \"cccf12d4-0554-4ea8-95f3-ddf4e73317eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"
Oct 30 13:08:32.628345 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.628236    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj6n7\" (UniqueName: \"kubernetes.io/projected/855e2bc2-43d6-4b6a-827f-a778178d6e23-kube-api-access-xj6n7\") pod \"payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6\" (UID: \"855e2bc2-43d6-4b6a-827f-a778178d6e23\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"
Oct 30 13:08:32.694294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.688088    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br2jr\" (UniqueName: \"kubernetes.io/projected/cccf12d4-0554-4ea8-95f3-ddf4e73317eb-kube-api-access-br2jr\") pod \"payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n\" (UID: \"cccf12d4-0554-4ea8-95f3-ddf4e73317eb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"
Oct 30 13:08:32.733906 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.733875    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"
Oct 30 13:08:32.922451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:32.922423    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"
Oct 30 13:08:33.100789 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:33.100764    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"]
Oct 30 13:08:33.279917 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:08:33.279878    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855e2bc2_43d6_4b6a_827f_a778178d6e23.slice/crio-7bf16658f7dc55d0e9a882191d0982f26038f90508b62dade31d74a9eb31fdac WatchSource:0}: Error finding container 7bf16658f7dc55d0e9a882191d0982f26038f90508b62dade31d74a9eb31fdac: Status 404 returned error can't find the container with id 7bf16658f7dc55d0e9a882191d0982f26038f90508b62dade31d74a9eb31fdac
Oct 30 13:08:33.280202 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:33.280174    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"]
Oct 30 13:08:33.561292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:33.560813    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n" event={"ID":"cccf12d4-0554-4ea8-95f3-ddf4e73317eb","Type":"ContainerStarted","Data":"13fc22bd7cf83bf3b284c2cbfb78b7df10ebc66fec3b06eda29a8352c5925d5b"}
Oct 30 13:08:33.561292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:33.560886    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n" event={"ID":"cccf12d4-0554-4ea8-95f3-ddf4e73317eb","Type":"ContainerStarted","Data":"6f576b3d09cff0a4197249d9c0fa16aa1c5ea1ce30d37ad4596fef3b3ab7fd2d"}
Oct 30 13:08:33.561850 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:33.561422    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"
Oct 30 13:08:33.565182 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:33.565152    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6" event={"ID":"855e2bc2-43d6-4b6a-827f-a778178d6e23","Type":"ContainerStarted","Data":"d2d63accbeb2d6150c235f3a0f341989042f24d0ee9729eabb1c03b35193ded7"}
Oct 30 13:08:33.566672 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:33.566647    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6" event={"ID":"855e2bc2-43d6-4b6a-827f-a778178d6e23","Type":"ContainerStarted","Data":"7bf16658f7dc55d0e9a882191d0982f26038f90508b62dade31d74a9eb31fdac"}
Oct 30 13:08:33.567714 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:33.567695    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"
Oct 30 13:08:33.612204 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:33.612054    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n" podStartSLOduration=1.61203433 podStartE2EDuration="1.61203433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:33.609674051 +0000 UTC m=+1355.193103775" watchObservedRunningTime="2026-10-30 13:08:33.61203433 +0000 UTC m=+1355.195464020"
Oct 30 13:08:33.661402 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:33.661350    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6" podStartSLOduration=1.661334533 podStartE2EDuration="1.661334533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:33.657322426 +0000 UTC m=+1355.240752139" watchObservedRunningTime="2026-10-30 13:08:33.661334533 +0000 UTC m=+1355.244764219"
Oct 30 13:08:39.438180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:39.438099    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"
Oct 30 13:08:44.578487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:44.578456    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"
Oct 30 13:08:45.585053 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:45.585024    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"
Oct 30 13:08:48.649303 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:08:48.649252    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:08:52.032778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.032745    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"]
Oct 30 13:08:52.037369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.037346    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:52.051944 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.051913    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d55cc4a-7c7b-4171-82f6-f274324b2228-metrics-certs\") pod \"maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:52.052098 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.052050    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d55cc4a-7c7b-4171-82f6-f274324b2228-maas-api-tls\") pod \"maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:52.052222 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.052181    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d55cc4a-7c7b-4171-82f6-f274324b2228-openshift-service-ca\") pod \"maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:52.052382 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.052341    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q9cr\" (UniqueName: \"kubernetes.io/projected/8d55cc4a-7c7b-4171-82f6-f274324b2228-kube-api-access-6q9cr\") pod \"maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:52.063209 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.063180    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-17fc433e-serving-cert\""
Oct 30 13:08:52.084357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.084321    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"]
Oct 30 13:08:52.153344 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.153306    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q9cr\" (UniqueName: \"kubernetes.io/projected/8d55cc4a-7c7b-4171-82f6-f274324b2228-kube-api-access-6q9cr\") pod \"maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:52.153527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.153465    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d55cc4a-7c7b-4171-82f6-f274324b2228-metrics-certs\") pod \"maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:52.153589 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.153534    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d55cc4a-7c7b-4171-82f6-f274324b2228-maas-api-tls\") pod \"maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:52.153631 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.153587    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d55cc4a-7c7b-4171-82f6-f274324b2228-openshift-service-ca\") pod \"maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:52.154348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.154324    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d55cc4a-7c7b-4171-82f6-f274324b2228-openshift-service-ca\") pod \"maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:52.156319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.156258    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d55cc4a-7c7b-4171-82f6-f274324b2228-metrics-certs\") pod \"maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:52.156406 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.156318    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d55cc4a-7c7b-4171-82f6-f274324b2228-maas-api-tls\") pod \"maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:52.180117 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.180080    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q9cr\" (UniqueName: \"kubernetes.io/projected/8d55cc4a-7c7b-4171-82f6-f274324b2228-kube-api-access-6q9cr\") pod \"maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:52.352404 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.352317    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:52.691230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.691197    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"]
Oct 30 13:08:52.692857 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:08:52.692829    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d55cc4a_7c7b_4171_82f6_f274324b2228.slice/crio-dffe7f97f43fe8c95de243f7656ae511e4ace6576408a856cf5ba275e60dd08f WatchSource:0}: Error finding container dffe7f97f43fe8c95de243f7656ae511e4ace6576408a856cf5ba275e60dd08f: Status 404 returned error can't find the container with id dffe7f97f43fe8c95de243f7656ae511e4ace6576408a856cf5ba275e60dd08f
Oct 30 13:08:52.772900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:52.772860    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk" event={"ID":"8d55cc4a-7c7b-4171-82f6-f274324b2228","Type":"ContainerStarted","Data":"dffe7f97f43fe8c95de243f7656ae511e4ace6576408a856cf5ba275e60dd08f"}
Oct 30 13:08:53.785420 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:53.785376    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk" event={"ID":"8d55cc4a-7c7b-4171-82f6-f274324b2228","Type":"ContainerStarted","Data":"cbd41e42df51561d6c4cf5759a7d6a74bd0d39fce549d9c8d2a7869c84bb2274"}
Oct 30 13:08:53.786683 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:53.786655    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:08:53.830260 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:53.830206    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk" podStartSLOduration=1.420700538 podStartE2EDuration="1.830190957s" totalImagesPullingTime="409.490419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:53.828908292 +0000 UTC m=+1375.412337999" watchObservedRunningTime="2026-10-30 13:08:53.830190957 +0000 UTC m=+1375.413620650"
Oct 30 13:08:56.619621 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:56.607988    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"]
Oct 30 13:08:56.624410 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:56.624378    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"
Oct 30 13:08:56.643080 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:56.643052    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"]
Oct 30 13:08:56.657387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:56.648120    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20b26c6-2e02-47d3-89e7-c4ca3b34da85-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw\" (UID: \"d20b26c6-2e02-47d3-89e7-c4ca3b34da85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"
Oct 30 13:08:56.657387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:56.648213    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqhj6\" (UniqueName: \"kubernetes.io/projected/d20b26c6-2e02-47d3-89e7-c4ca3b34da85-kube-api-access-jqhj6\") pod \"payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw\" (UID: \"d20b26c6-2e02-47d3-89e7-c4ca3b34da85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"
Oct 30 13:08:56.698391 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:56.685564    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-0e874e\""
Oct 30 13:08:56.716415 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:56.706643    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-0e874e-dockercfg-xdvdm\""
Oct 30 13:08:56.798560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:56.798521    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20b26c6-2e02-47d3-89e7-c4ca3b34da85-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw\" (UID: \"d20b26c6-2e02-47d3-89e7-c4ca3b34da85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"
Oct 30 13:08:56.798758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:56.798623    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqhj6\" (UniqueName: \"kubernetes.io/projected/d20b26c6-2e02-47d3-89e7-c4ca3b34da85-kube-api-access-jqhj6\") pod \"payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw\" (UID: \"d20b26c6-2e02-47d3-89e7-c4ca3b34da85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"
Oct 30 13:08:56.831296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:56.801426    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20b26c6-2e02-47d3-89e7-c4ca3b34da85-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw\" (UID: \"d20b26c6-2e02-47d3-89e7-c4ca3b34da85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"
Oct 30 13:08:56.915198 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:56.915119    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"]
Oct 30 13:08:56.940438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:56.940403    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"]
Oct 30 13:08:56.940718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:56.932294    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"
Oct 30 13:08:57.001613 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.001587    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqhj6\" (UniqueName: \"kubernetes.io/projected/d20b26c6-2e02-47d3-89e7-c4ca3b34da85-kube-api-access-jqhj6\") pod \"payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw\" (UID: \"d20b26c6-2e02-47d3-89e7-c4ca3b34da85\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"
Oct 30 13:08:57.007633 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.007598    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478ff6ac-9262-49aa-b3cc-34423527b462-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z\" (UID: \"478ff6ac-9262-49aa-b3cc-34423527b462\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"
Oct 30 13:08:57.007824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.007764    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vchdp\" (UniqueName: \"kubernetes.io/projected/478ff6ac-9262-49aa-b3cc-34423527b462-kube-api-access-vchdp\") pod \"payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z\" (UID: \"478ff6ac-9262-49aa-b3cc-34423527b462\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"
Oct 30 13:08:57.008867 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.008843    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"
Oct 30 13:08:57.109479 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.109170    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478ff6ac-9262-49aa-b3cc-34423527b462-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z\" (UID: \"478ff6ac-9262-49aa-b3cc-34423527b462\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"
Oct 30 13:08:57.109479 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.109364    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vchdp\" (UniqueName: \"kubernetes.io/projected/478ff6ac-9262-49aa-b3cc-34423527b462-kube-api-access-vchdp\") pod \"payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z\" (UID: \"478ff6ac-9262-49aa-b3cc-34423527b462\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"
Oct 30 13:08:57.109983 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.109958    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478ff6ac-9262-49aa-b3cc-34423527b462-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z\" (UID: \"478ff6ac-9262-49aa-b3cc-34423527b462\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"
Oct 30 13:08:57.132533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.132501    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vchdp\" (UniqueName: \"kubernetes.io/projected/478ff6ac-9262-49aa-b3cc-34423527b462-kube-api-access-vchdp\") pod \"payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z\" (UID: \"478ff6ac-9262-49aa-b3cc-34423527b462\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"
Oct 30 13:08:57.347397 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.347367    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"]
Oct 30 13:08:57.347894 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:08:57.347870    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20b26c6_2e02_47d3_89e7_c4ca3b34da85.slice/crio-a4377958cc8591296b721147004511dea541542114f2ae7ba372b5f98666aeb1 WatchSource:0}: Error finding container a4377958cc8591296b721147004511dea541542114f2ae7ba372b5f98666aeb1: Status 404 returned error can't find the container with id a4377958cc8591296b721147004511dea541542114f2ae7ba372b5f98666aeb1
Oct 30 13:08:57.369422 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.369376    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"
Oct 30 13:08:57.689288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.689243    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"]
Oct 30 13:08:57.690952 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:08:57.690903    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478ff6ac_9262_49aa_b3cc_34423527b462.slice/crio-582787d173fc3b5fdaeec5a1c13a2d8164c54284eb9e350cc6896105cec659eb WatchSource:0}: Error finding container 582787d173fc3b5fdaeec5a1c13a2d8164c54284eb9e350cc6896105cec659eb: Status 404 returned error can't find the container with id 582787d173fc3b5fdaeec5a1c13a2d8164c54284eb9e350cc6896105cec659eb
Oct 30 13:08:57.873590 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.873556    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z" event={"ID":"478ff6ac-9262-49aa-b3cc-34423527b462","Type":"ContainerStarted","Data":"16ba6f037c674c54513c7ba81c720bed8cac21cd7abf5f8598ee42cc8c55e46e"}
Oct 30 13:08:57.873794 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.873618    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z" event={"ID":"478ff6ac-9262-49aa-b3cc-34423527b462","Type":"ContainerStarted","Data":"582787d173fc3b5fdaeec5a1c13a2d8164c54284eb9e350cc6896105cec659eb"}
Oct 30 13:08:57.873794 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.873650    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"
Oct 30 13:08:57.875318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.875255    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw" event={"ID":"d20b26c6-2e02-47d3-89e7-c4ca3b34da85","Type":"ContainerStarted","Data":"2b607f697dae92f011563370974a3756437387be945b56b46b8fd3702b4a8c31"}
Oct 30 13:08:57.875420 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.875336    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw" event={"ID":"d20b26c6-2e02-47d3-89e7-c4ca3b34da85","Type":"ContainerStarted","Data":"a4377958cc8591296b721147004511dea541542114f2ae7ba372b5f98666aeb1"}
Oct 30 13:08:57.875483 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.875430    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"
Oct 30 13:08:57.919332 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.919235    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z" podStartSLOduration=1.919219872 podStartE2EDuration="1.919219872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:57.915507235 +0000 UTC m=+1379.498936939" watchObservedRunningTime="2026-10-30 13:08:57.919219872 +0000 UTC m=+1379.502649564"
Oct 30 13:08:57.958305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:57.958195    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw" podStartSLOduration=1.9581760209999999 podStartE2EDuration="1.958176021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:57.955719114 +0000 UTC m=+1379.539148878" watchObservedRunningTime="2026-10-30 13:08:57.958176021 +0000 UTC m=+1379.541605715"
Oct 30 13:08:59.449616 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.449574    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"]
Oct 30 13:08:59.460656 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.460620    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:08:59.480735 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.480706    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"]
Oct 30 13:08:59.493118 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.493085    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-0b2c92-serving-cert\""
Oct 30 13:08:59.551373 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.551337    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6311bfd1-218e-4fe6-a79c-74f3d127e805-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:08:59.551601 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.551543    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6311bfd1-218e-4fe6-a79c-74f3d127e805-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:08:59.551601 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.551588    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h69g5\" (UniqueName: \"kubernetes.io/projected/6311bfd1-218e-4fe6-a79c-74f3d127e805-kube-api-access-h69g5\") pod \"maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:08:59.551842 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.551828    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6311bfd1-218e-4fe6-a79c-74f3d127e805-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:08:59.652997 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.652964    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6311bfd1-218e-4fe6-a79c-74f3d127e805-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:08:59.653210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.653010    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6311bfd1-218e-4fe6-a79c-74f3d127e805-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:08:59.653210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.653109    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6311bfd1-218e-4fe6-a79c-74f3d127e805-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:08:59.653210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.653142    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h69g5\" (UniqueName: \"kubernetes.io/projected/6311bfd1-218e-4fe6-a79c-74f3d127e805-kube-api-access-h69g5\") pod \"maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:08:59.653909 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.653835    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6311bfd1-218e-4fe6-a79c-74f3d127e805-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:08:59.669453 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.657527    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6311bfd1-218e-4fe6-a79c-74f3d127e805-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:08:59.669453 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.657980    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6311bfd1-218e-4fe6-a79c-74f3d127e805-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:08:59.678265 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.678232    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h69g5\" (UniqueName: \"kubernetes.io/projected/6311bfd1-218e-4fe6-a79c-74f3d127e805-kube-api-access-h69g5\") pod \"maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:08:59.778549 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:08:59.778516    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:09:00.113191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:00.113163    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"]
Oct 30 13:09:00.114813 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:09:00.114782    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6311bfd1_218e_4fe6_a79c_74f3d127e805.slice/crio-36453df53695b065efed2ad44044bec250ab69ec213d92b73e720ad30c4d01ac WatchSource:0}: Error finding container 36453df53695b065efed2ad44044bec250ab69ec213d92b73e720ad30c4d01ac: Status 404 returned error can't find the container with id 36453df53695b065efed2ad44044bec250ab69ec213d92b73e720ad30c4d01ac
Oct 30 13:09:00.819524 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:00.819453    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:09:01.057919 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:01.057857    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c" event={"ID":"6311bfd1-218e-4fe6-a79c-74f3d127e805","Type":"ContainerStarted","Data":"3a8fcbd6158c96b6a981624a83de24395ab08d27dc90da343f299fa766b6101d"}
Oct 30 13:09:01.058111 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:01.057927    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c" event={"ID":"6311bfd1-218e-4fe6-a79c-74f3d127e805","Type":"ContainerStarted","Data":"36453df53695b065efed2ad44044bec250ab69ec213d92b73e720ad30c4d01ac"}
Oct 30 13:09:01.059139 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:01.059116    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:09:01.101233 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:01.101124    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c" podStartSLOduration=1.733246485 podStartE2EDuration="2.101106704s" totalImagesPullingTime="367.860219ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:01.09975928 +0000 UTC m=+1382.683189004" watchObservedRunningTime="2026-10-30 13:09:01.101106704 +0000 UTC m=+1382.684536397"
Oct 30 13:09:08.106307 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:08.106100    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:09:08.904979 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:08.904948    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"
Oct 30 13:09:08.905586 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:08.905567    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"
Oct 30 13:09:18.292817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:18.292783    5893 generic.go:356] "Generic (PLEG): container finished" podID="abf058d4-f482-47b8-8610-6491dfd318c2" containerID="eaadaa18fa1d64fd72ea8d05ffb70b378363f5e2dfe40038e30dce43fe16ca6a" exitCode=0
Oct 30 13:09:18.293362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:18.292859    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85" event={"ID":"abf058d4-f482-47b8-8610-6491dfd318c2","Type":"ContainerDied","Data":"eaadaa18fa1d64fd72ea8d05ffb70b378363f5e2dfe40038e30dce43fe16ca6a"}
Oct 30 13:09:18.293443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:18.293364    5893 scope.go:122] "RemoveContainer" containerID="eaadaa18fa1d64fd72ea8d05ffb70b378363f5e2dfe40038e30dce43fe16ca6a"
Oct 30 13:09:18.294864 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:18.294843    5893 generic.go:356] "Generic (PLEG): container finished" podID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerID="0fc25ddcb9028983f0fa28caede55b2289399949b584b34089f7e4f75e2f1763" exitCode=0
Oct 30 13:09:18.294948 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:18.294915    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7" event={"ID":"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7","Type":"ContainerDied","Data":"0fc25ddcb9028983f0fa28caede55b2289399949b584b34089f7e4f75e2f1763"}
Oct 30 13:09:18.295318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:18.295303    5893 scope.go:122] "RemoveContainer" containerID="0fc25ddcb9028983f0fa28caede55b2289399949b584b34089f7e4f75e2f1763"
Oct 30 13:09:18.295446 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:18.295418    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:09:18.295509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:18.295493    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:09:18.295543 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:18.295535    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:09:18.296799 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:18.296770    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:09:18.296898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:18.296832    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:09:18.296898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:18.296866    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:09:18.656593 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:09:18.656484    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:09:19.309000 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:19.308962    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85" event={"ID":"abf058d4-f482-47b8-8610-6491dfd318c2","Type":"ContainerStarted","Data":"3914cf6f21d2753ec2dd310233301b49d94968f5b70859534538a7465d1aa878"}
Oct 30 13:09:19.311128 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:19.311106    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7" event={"ID":"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7","Type":"ContainerStarted","Data":"a022a76d332d34826321a5583f4fc8dd29b13e1df2b48af9ce86a9990dec8a0d"}
Oct 30 13:09:19.914527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:19.914321    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:09:19.914901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:19.914873    5893 patch_prober.go:29] interesting pod/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 13:09:19.915128 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:19.915096    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7" podUID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 13:09:20.218129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:20.218100    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:09:20.218532 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:20.218510    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:09:20.218663 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:20.218556    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85" podUID="abf058d4-f482-47b8-8610-6491dfd318c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 13:09:20.913145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:20.913114    5893 patch_prober.go:29] interesting pod/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 13:09:20.913613 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:20.913172    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7" podUID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 13:09:21.218651 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:21.218609    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:09:21.218851 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:21.218669    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85" podUID="abf058d4-f482-47b8-8610-6491dfd318c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 13:09:21.913224 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:21.913187    5893 patch_prober.go:29] interesting pod/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 13:09:21.913741 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:21.913252    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7" podUID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 13:09:22.218542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:22.218504    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:09:22.218739 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:22.218567    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85" podUID="abf058d4-f482-47b8-8610-6491dfd318c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 13:09:22.913519 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:22.913468    5893 patch_prober.go:29] interesting pod/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 13:09:22.914151 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:22.914122    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7" podUID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 13:09:23.204432 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.204347    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"]
Oct 30 13:09:23.210181 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.210153    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"
Oct 30 13:09:23.210928 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.210860    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"]
Oct 30 13:09:23.219201 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.219173    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:09:23.219467 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.219438    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85" podUID="abf058d4-f482-47b8-8610-6491dfd318c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 13:09:23.224951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.224922    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-75abce\""
Oct 30 13:09:23.230527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.230495    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsv42\" (UniqueName: \"kubernetes.io/projected/9395056f-a35b-457d-9c77-3877c5571e39-kube-api-access-dsv42\") pod \"payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn\" (UID: \"9395056f-a35b-457d-9c77-3877c5571e39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"
Oct 30 13:09:23.230844 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.230826    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9395056f-a35b-457d-9c77-3877c5571e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn\" (UID: \"9395056f-a35b-457d-9c77-3877c5571e39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"
Oct 30 13:09:23.236492 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.236469    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-75abce-dockercfg-srqhl\""
Oct 30 13:09:23.409297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.406415    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:09:23.434300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.424361    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9395056f-a35b-457d-9c77-3877c5571e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn\" (UID: \"9395056f-a35b-457d-9c77-3877c5571e39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"
Oct 30 13:09:23.434300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.424520    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsv42\" (UniqueName: \"kubernetes.io/projected/9395056f-a35b-457d-9c77-3877c5571e39-kube-api-access-dsv42\") pod \"payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn\" (UID: \"9395056f-a35b-457d-9c77-3877c5571e39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"
Oct 30 13:09:23.434300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.426245    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9395056f-a35b-457d-9c77-3877c5571e39-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn\" (UID: \"9395056f-a35b-457d-9c77-3877c5571e39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"
Oct 30 13:09:23.746305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.746251    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsv42\" (UniqueName: \"kubernetes.io/projected/9395056f-a35b-457d-9c77-3877c5571e39-kube-api-access-dsv42\") pod \"payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn\" (UID: \"9395056f-a35b-457d-9c77-3877c5571e39\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"
Oct 30 13:09:23.832593 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.832566    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"
Oct 30 13:09:23.915382 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.914884    5893 patch_prober.go:29] interesting pod/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 13:09:23.915382 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:23.914965    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7" podUID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 13:09:24.219830 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:24.219788    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:09:24.220033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:24.219862    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85" podUID="abf058d4-f482-47b8-8610-6491dfd318c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Oct 30 13:09:24.390374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:24.373610    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"]
Oct 30 13:09:24.659814 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:24.659722    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn" event={"ID":"9395056f-a35b-457d-9c77-3877c5571e39","Type":"ContainerStarted","Data":"24a177731affc25bed46bb98d0a048da3badd0403cd31a5f912fc689e740caff"}
Oct 30 13:09:24.881299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:24.881111    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"]
Oct 30 13:09:24.888754 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:24.888727    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:24.906185 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:24.905299    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-0e874e-serving-cert\""
Oct 30 13:09:24.913331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:24.913237    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"]
Oct 30 13:09:24.916584 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:24.916076    5893 patch_prober.go:29] interesting pod/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Oct 30 13:09:24.916584 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:24.916128    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7" podUID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Oct 30 13:09:24.977541 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:24.977509    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce692a70-8f35-4154-a801-c53be256f2f3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:24.977870 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:24.977585    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce692a70-8f35-4154-a801-c53be256f2f3-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:24.977870 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:24.977840    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce692a70-8f35-4154-a801-c53be256f2f3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:24.977990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:24.977970    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mpfc\" (UniqueName: \"kubernetes.io/projected/ce692a70-8f35-4154-a801-c53be256f2f3-kube-api-access-8mpfc\") pod \"maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:25.079765 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.079723    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce692a70-8f35-4154-a801-c53be256f2f3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:25.079998 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.079842    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce692a70-8f35-4154-a801-c53be256f2f3-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:25.080074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.080002    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce692a70-8f35-4154-a801-c53be256f2f3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:25.080138 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.080096    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mpfc\" (UniqueName: \"kubernetes.io/projected/ce692a70-8f35-4154-a801-c53be256f2f3-kube-api-access-8mpfc\") pod \"maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:25.080778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.080618    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce692a70-8f35-4154-a801-c53be256f2f3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:25.084047 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.083980    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce692a70-8f35-4154-a801-c53be256f2f3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:25.085106 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.085070    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce692a70-8f35-4154-a801-c53be256f2f3-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:25.104048 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.104009    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mpfc\" (UniqueName: \"kubernetes.io/projected/ce692a70-8f35-4154-a801-c53be256f2f3-kube-api-access-8mpfc\") pod \"maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:25.221336 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.221091    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:25.307265 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.305642    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:09:25.308980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.308897    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:09:25.310372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.310339    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:09:25.647628 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.647595    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"]
Oct 30 13:09:25.649891 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:09:25.649860    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce692a70_8f35_4154_a801_c53be256f2f3.slice/crio-d71e77c2fb8e3818c2c031ee71cf002a0aa8afe51632c884ea200155d43e7a26 WatchSource:0}: Error finding container d71e77c2fb8e3818c2c031ee71cf002a0aa8afe51632c884ea200155d43e7a26: Status 404 returned error can't find the container with id d71e77c2fb8e3818c2c031ee71cf002a0aa8afe51632c884ea200155d43e7a26
Oct 30 13:09:25.650575 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.650546    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn" event={"ID":"9395056f-a35b-457d-9c77-3877c5571e39","Type":"ContainerStarted","Data":"d22b05b0689bea4202fd013086ac944009786b90c83f53a1f4e2752f1aa7a7f8"}
Oct 30 13:09:25.650912 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.650890    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"
Oct 30 13:09:25.711934 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.711878    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn" podStartSLOduration=2.711860697 podStartE2EDuration="2.711860697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:25.710071087 +0000 UTC m=+1407.293500777" watchObservedRunningTime="2026-10-30 13:09:25.711860697 +0000 UTC m=+1407.295290374"
Oct 30 13:09:25.937919 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.937827    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:09:25.941777 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:25.941751    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:09:26.660238 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:26.660201    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7" event={"ID":"ce692a70-8f35-4154-a801-c53be256f2f3","Type":"ContainerStarted","Data":"5ff4fec9798e939547b3a18a3e6b1c5dc25ee53f0d91bcd1f1acdb210c3f7513"}
Oct 30 13:09:26.660448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:26.660284    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7" event={"ID":"ce692a70-8f35-4154-a801-c53be256f2f3","Type":"ContainerStarted","Data":"d71e77c2fb8e3818c2c031ee71cf002a0aa8afe51632c884ea200155d43e7a26"}
Oct 30 13:09:26.660692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:26.660669    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:26.706902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:26.706843    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7" podStartSLOduration=2.408092887 podStartE2EDuration="2.706822746s" totalImagesPullingTime="298.729859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:26.703810357 +0000 UTC m=+1408.287240069" watchObservedRunningTime="2026-10-30 13:09:26.706822746 +0000 UTC m=+1408.290252440"
Oct 30 13:09:32.700592 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:32.700562    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:09:36.664114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:36.664087    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"
Oct 30 13:09:41.113753 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:41.113713    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"]
Oct 30 13:09:41.114243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:41.113919    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7" podUID="54d53f0f-efc7-43b4-86eb-0e105ea50355" containerName="istio-proxy" containerID="cri-o://258a98388a7b7ffb9881f24e22802c0efed2a9180adfd984bf84c469cd7ab482" gracePeriod=30
Oct 30 13:09:44.809094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:44.809062    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"]
Oct 30 13:09:44.812798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:44.812766    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"
Oct 30 13:09:44.842540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:44.842509    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-3e8ee4\""
Oct 30 13:09:44.848984 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:44.848951    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"]
Oct 30 13:09:44.896760 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:44.896709    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"]
Oct 30 13:09:44.915139 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:44.913772    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vvhn\" (UniqueName: \"kubernetes.io/projected/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb-kube-api-access-9vvhn\") pod \"payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj\" (UID: \"f2e21998-d091-4ca7-87d6-3dcefdfaa4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"
Oct 30 13:09:44.915139 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:44.914027    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj\" (UID: \"f2e21998-d091-4ca7-87d6-3dcefdfaa4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"
Oct 30 13:09:44.916144 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:44.915587    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"]
Oct 30 13:09:44.916144 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:44.915694    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"
Oct 30 13:09:44.937633 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:44.937578    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-3e8ee4-dockercfg-jrkdg\""
Oct 30 13:09:45.015437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.015391    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00cbb825-5817-49ad-87b0-7f7d8990facb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8\" (UID: \"00cbb825-5817-49ad-87b0-7f7d8990facb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"
Oct 30 13:09:45.015615 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.015457    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj\" (UID: \"f2e21998-d091-4ca7-87d6-3dcefdfaa4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"
Oct 30 13:09:45.015615 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.015543    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjw9j\" (UniqueName: \"kubernetes.io/projected/00cbb825-5817-49ad-87b0-7f7d8990facb-kube-api-access-cjw9j\") pod \"payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8\" (UID: \"00cbb825-5817-49ad-87b0-7f7d8990facb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"
Oct 30 13:09:45.015728 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.015713    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vvhn\" (UniqueName: \"kubernetes.io/projected/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb-kube-api-access-9vvhn\") pod \"payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj\" (UID: \"f2e21998-d091-4ca7-87d6-3dcefdfaa4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"
Oct 30 13:09:45.016966 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.016939    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj\" (UID: \"f2e21998-d091-4ca7-87d6-3dcefdfaa4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"
Oct 30 13:09:45.070339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.070162    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vvhn\" (UniqueName: \"kubernetes.io/projected/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb-kube-api-access-9vvhn\") pod \"payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj\" (UID: \"f2e21998-d091-4ca7-87d6-3dcefdfaa4fb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"
Oct 30 13:09:45.122211 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.118345    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00cbb825-5817-49ad-87b0-7f7d8990facb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8\" (UID: \"00cbb825-5817-49ad-87b0-7f7d8990facb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"
Oct 30 13:09:45.122211 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.118449    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjw9j\" (UniqueName: \"kubernetes.io/projected/00cbb825-5817-49ad-87b0-7f7d8990facb-kube-api-access-cjw9j\") pod \"payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8\" (UID: \"00cbb825-5817-49ad-87b0-7f7d8990facb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"
Oct 30 13:09:45.122211 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.119729    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00cbb825-5817-49ad-87b0-7f7d8990facb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8\" (UID: \"00cbb825-5817-49ad-87b0-7f7d8990facb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"
Oct 30 13:09:45.126489 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.125668    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"
Oct 30 13:09:45.147734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.147661    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjw9j\" (UniqueName: \"kubernetes.io/projected/00cbb825-5817-49ad-87b0-7f7d8990facb-kube-api-access-cjw9j\") pod \"payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8\" (UID: \"00cbb825-5817-49ad-87b0-7f7d8990facb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"
Oct 30 13:09:45.291363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.290525    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"
Oct 30 13:09:45.470212 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.470184    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"]
Oct 30 13:09:45.473290 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:09:45.473246    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e21998_d091_4ca7_87d6_3dcefdfaa4fb.slice/crio-e324a06fea4a7280de252819772b8a535a8fe5bc0384be4c8802b3fcab4474b9 WatchSource:0}: Error finding container e324a06fea4a7280de252819772b8a535a8fe5bc0384be4c8802b3fcab4474b9: Status 404 returned error can't find the container with id e324a06fea4a7280de252819772b8a535a8fe5bc0384be4c8802b3fcab4474b9
Oct 30 13:09:45.631370 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.631341    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"]
Oct 30 13:09:45.632702 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:09:45.632680    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00cbb825_5817_49ad_87b0_7f7d8990facb.slice/crio-075b9c301adef3d6fddbb8b3d409cbf3b23835d7dae83ce699cb68cfa491f4a9 WatchSource:0}: Error finding container 075b9c301adef3d6fddbb8b3d409cbf3b23835d7dae83ce699cb68cfa491f4a9: Status 404 returned error can't find the container with id 075b9c301adef3d6fddbb8b3d409cbf3b23835d7dae83ce699cb68cfa491f4a9
Oct 30 13:09:45.930294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.930174    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj" event={"ID":"f2e21998-d091-4ca7-87d6-3dcefdfaa4fb","Type":"ContainerStarted","Data":"6a985c1e2697ef9a84d5e8ac9c5acd6998beabab1cded27cae35448ebd62994b"}
Oct 30 13:09:45.930294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.930238    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj" event={"ID":"f2e21998-d091-4ca7-87d6-3dcefdfaa4fb","Type":"ContainerStarted","Data":"e324a06fea4a7280de252819772b8a535a8fe5bc0384be4c8802b3fcab4474b9"}
Oct 30 13:09:45.930794 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.930416    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"
Oct 30 13:09:45.931984 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.931956    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8" event={"ID":"00cbb825-5817-49ad-87b0-7f7d8990facb","Type":"ContainerStarted","Data":"43f7a4662d3deac04ac3a873252b358a90a9dff2ffbedd96b40ae823b52a33c3"}
Oct 30 13:09:45.932072 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.931990    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8" event={"ID":"00cbb825-5817-49ad-87b0-7f7d8990facb","Type":"ContainerStarted","Data":"075b9c301adef3d6fddbb8b3d409cbf3b23835d7dae83ce699cb68cfa491f4a9"}
Oct 30 13:09:45.932115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.932080    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"
Oct 30 13:09:45.973786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:45.973733    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj" podStartSLOduration=1.973716412 podStartE2EDuration="1.973716412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:45.969378032 +0000 UTC m=+1427.552807726" watchObservedRunningTime="2026-10-30 13:09:45.973716412 +0000 UTC m=+1427.557146104"
Oct 30 13:09:46.017469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.017423    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8" podStartSLOduration=2.017407736 podStartE2EDuration="2.017407736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:46.015298208 +0000 UTC m=+1427.598727895" watchObservedRunningTime="2026-10-30 13:09:46.017407736 +0000 UTC m=+1427.600837438"
Oct 30 13:09:46.429454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.429417    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:09:46.543823 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.543741    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54d53f0f-efc7-43b4-86eb-0e105ea50355-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d53f0f-efc7-43b4-86eb-0e105ea50355-istiod-ca-cert\") pod \"54d53f0f-efc7-43b4-86eb-0e105ea50355\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") "
Oct 30 13:09:46.544203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.544182    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-token\" (UniqueName: \"kubernetes.io/projected/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-token\") pod \"54d53f0f-efc7-43b4-86eb-0e105ea50355\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") "
Oct 30 13:09:46.544471 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.544452    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-podinfo\") pod \"54d53f0f-efc7-43b4-86eb-0e105ea50355\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") "
Oct 30 13:09:46.544627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.544612    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-envoy\") pod \"54d53f0f-efc7-43b4-86eb-0e105ea50355\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") "
Oct 30 13:09:46.544784 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.544759    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d53f0f-efc7-43b4-86eb-0e105ea50355-kube-api-access-pzpqz\" (UniqueName: \"kubernetes.io/projected/54d53f0f-efc7-43b4-86eb-0e105ea50355-kube-api-access-pzpqz\") pod \"54d53f0f-efc7-43b4-86eb-0e105ea50355\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") "
Oct 30 13:09:46.544936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.544821    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-data\") pod \"54d53f0f-efc7-43b4-86eb-0e105ea50355\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") "
Oct 30 13:09:46.544936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.544883    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-credential-socket\") pod \"54d53f0f-efc7-43b4-86eb-0e105ea50355\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") "
Oct 30 13:09:46.545099 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.544943    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-workload-certs\") pod \"54d53f0f-efc7-43b4-86eb-0e105ea50355\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") "
Oct 30 13:09:46.545099 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.544996    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-workload-socket\") pod \"54d53f0f-efc7-43b4-86eb-0e105ea50355\" (UID: \"54d53f0f-efc7-43b4-86eb-0e105ea50355\") "
Oct 30 13:09:46.545237 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.545142    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-data" pod "54d53f0f-efc7-43b4-86eb-0e105ea50355" (UID: "54d53f0f-efc7-43b4-86eb-0e105ea50355"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:46.545723 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.545692    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-workload-certs" pod "54d53f0f-efc7-43b4-86eb-0e105ea50355" (UID: "54d53f0f-efc7-43b4-86eb-0e105ea50355"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:46.545832 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.545812    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:09:46.545891 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.545835    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:09:46.546061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.546024    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-credential-socket" pod "54d53f0f-efc7-43b4-86eb-0e105ea50355" (UID: "54d53f0f-efc7-43b4-86eb-0e105ea50355"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:46.546061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.546040    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-workload-socket" pod "54d53f0f-efc7-43b4-86eb-0e105ea50355" (UID: "54d53f0f-efc7-43b4-86eb-0e105ea50355"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:46.546582 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.546561    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d53f0f-efc7-43b4-86eb-0e105ea50355-istiod-ca-cert" pod "54d53f0f-efc7-43b4-86eb-0e105ea50355" (UID: "54d53f0f-efc7-43b4-86eb-0e105ea50355"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:09:46.550005 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.549098    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-podinfo" pod "54d53f0f-efc7-43b4-86eb-0e105ea50355" (UID: "54d53f0f-efc7-43b4-86eb-0e105ea50355"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:09:46.550005 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.550549    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-token" pod "54d53f0f-efc7-43b4-86eb-0e105ea50355" (UID: "54d53f0f-efc7-43b4-86eb-0e105ea50355"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:46.564523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.552196    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-envoy" pod "54d53f0f-efc7-43b4-86eb-0e105ea50355" (UID: "54d53f0f-efc7-43b4-86eb-0e105ea50355"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:09:46.564523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.553420    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d53f0f-efc7-43b4-86eb-0e105ea50355-kube-api-access-pzpqz" pod "54d53f0f-efc7-43b4-86eb-0e105ea50355" (UID: "54d53f0f-efc7-43b4-86eb-0e105ea50355"). InnerVolumeSpecName "kube-api-access-pzpqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:09:46.648160 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.648049    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54d53f0f-efc7-43b4-86eb-0e105ea50355-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:09:46.648160 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.648086    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:09:46.648160 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.648100    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:09:46.648160 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.648120    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:09:46.648160 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.648136    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzpqz\" (UniqueName: \"kubernetes.io/projected/54d53f0f-efc7-43b4-86eb-0e105ea50355-kube-api-access-pzpqz\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:09:46.657702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.648620    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:09:46.657702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.648645    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54d53f0f-efc7-43b4-86eb-0e105ea50355-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:09:46.944417 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.944326    5893 generic.go:356] "Generic (PLEG): container finished" podID="54d53f0f-efc7-43b4-86eb-0e105ea50355" containerID="258a98388a7b7ffb9881f24e22802c0efed2a9180adfd984bf84c469cd7ab482" exitCode=0
Oct 30 13:09:46.944981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.944346    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7" event={"ID":"54d53f0f-efc7-43b4-86eb-0e105ea50355","Type":"ContainerDied","Data":"258a98388a7b7ffb9881f24e22802c0efed2a9180adfd984bf84c469cd7ab482"}
Oct 30 13:09:46.945105 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.944361    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"
Oct 30 13:09:46.945105 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.945074    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7" event={"ID":"54d53f0f-efc7-43b4-86eb-0e105ea50355","Type":"ContainerDied","Data":"6744914a1f71f39ef3d6308b2cb581d841b61cd1aa13e6e1d6e07baf33ffa905"}
Oct 30 13:09:46.945216 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.945115    5893 scope.go:122] "RemoveContainer" containerID="258a98388a7b7ffb9881f24e22802c0efed2a9180adfd984bf84c469cd7ab482"
Oct 30 13:09:46.963137 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.963092    5893 scope.go:122] "RemoveContainer" containerID="258a98388a7b7ffb9881f24e22802c0efed2a9180adfd984bf84c469cd7ab482"
Oct 30 13:09:46.963505 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:09:46.963474    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258a98388a7b7ffb9881f24e22802c0efed2a9180adfd984bf84c469cd7ab482\": container with ID starting with 258a98388a7b7ffb9881f24e22802c0efed2a9180adfd984bf84c469cd7ab482 not found: ID does not exist" containerID="258a98388a7b7ffb9881f24e22802c0efed2a9180adfd984bf84c469cd7ab482"
Oct 30 13:09:46.963608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.963516    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258a98388a7b7ffb9881f24e22802c0efed2a9180adfd984bf84c469cd7ab482"} err="failed to get container status \"258a98388a7b7ffb9881f24e22802c0efed2a9180adfd984bf84c469cd7ab482\": rpc error: code = NotFound desc = could not find container \"258a98388a7b7ffb9881f24e22802c0efed2a9180adfd984bf84c469cd7ab482\": container with ID starting with 258a98388a7b7ffb9881f24e22802c0efed2a9180adfd984bf84c469cd7ab482 not found: ID does not exist"
Oct 30 13:09:46.999734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:46.999704    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"]
Oct 30 13:09:47.012237 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:47.012210    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-x2hw7"]
Oct 30 13:09:48.540042 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:48.540000    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54d53f0f-efc7-43b4-86eb-0e105ea50355" path="/var/lib/kubelet/pods/54d53f0f-efc7-43b4-86eb-0e105ea50355/volumes"
Oct 30 13:09:48.647554 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:09:48.647521    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:09:56.949586 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:56.949543    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"
Oct 30 13:09:56.950085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:09:56.949628    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"
Oct 30 13:10:02.832293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:02.832243    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"]
Oct 30 13:10:02.871828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:02.871798    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"]
Oct 30 13:10:06.025674 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.025643    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"]
Oct 30 13:10:06.026446 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.026430    5893 cpu_manager.go:414] "containerMap: removing container" podUID="54d53f0f-efc7-43b4-86eb-0e105ea50355" containerName="istio-proxy"
Oct 30 13:10:06.026502 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.026447    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d53f0f-efc7-43b4-86eb-0e105ea50355" containerName="istio-proxy"
Oct 30 13:10:06.026635 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.026625    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d53f0f-efc7-43b4-86eb-0e105ea50355" containerName="istio-proxy"
Oct 30 13:10:06.032211 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.032190    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.046743 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.046708    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1c0e6c42-openshift-default-dockercfg-zstsq\""
Oct 30 13:10:06.064462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.064434    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"]
Oct 30 13:10:06.132777 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.132731    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023728ea-7d4e-4d33-9618-320d46770bbd-istiod-ca-cert\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.133063 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.133041    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-istio-data\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.133202 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.133186    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-workload-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.133563 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.133523    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-workload-certs\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.133766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.133748    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023728ea-7d4e-4d33-9618-320d46770bbd-istio-token\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.135377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.135343    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9cm\" (UniqueName: \"kubernetes.io/projected/023728ea-7d4e-4d33-9618-320d46770bbd-kube-api-access-8n9cm\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.139928 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.135601    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-istio-envoy\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.140087 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.140067    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023728ea-7d4e-4d33-9618-320d46770bbd-istio-podinfo\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.140225 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.140208    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-credential-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.244295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.242464    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-workload-certs\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.244295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.242591    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023728ea-7d4e-4d33-9618-320d46770bbd-istio-token\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.244295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.242666    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9cm\" (UniqueName: \"kubernetes.io/projected/023728ea-7d4e-4d33-9618-320d46770bbd-kube-api-access-8n9cm\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.244295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.242715    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-istio-envoy\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.244295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.242745    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023728ea-7d4e-4d33-9618-320d46770bbd-istio-podinfo\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.244295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.242780    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-credential-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.244295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.242845    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023728ea-7d4e-4d33-9618-320d46770bbd-istiod-ca-cert\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.244295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.242889    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-istio-data\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.244295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.242930    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-workload-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.244295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.243601    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-workload-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.244295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.243910    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-workload-certs\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.262350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.245321    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-credential-socket\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.262350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.246262    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-istio-data\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.262350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.246932    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023728ea-7d4e-4d33-9618-320d46770bbd-istiod-ca-cert\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.277725 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.273300    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9cm\" (UniqueName: \"kubernetes.io/projected/023728ea-7d4e-4d33-9618-320d46770bbd-kube-api-access-8n9cm\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.312293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.306897    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023728ea-7d4e-4d33-9618-320d46770bbd-istio-podinfo\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.319742 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.316878    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023728ea-7d4e-4d33-9618-320d46770bbd-istio-token\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.324302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.324229    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-istio-envoy\") pod \"e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") " pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.350423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.350039    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:06.753047 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.752997    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"]
Oct 30 13:10:06.762324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.760086    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:06.762324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.760184    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:06.762324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:06.760230    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:10:07.132671 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:07.132578    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf" event={"ID":"023728ea-7d4e-4d33-9618-320d46770bbd","Type":"ContainerStarted","Data":"58b27ecd451d82105d9740b06ebdd2f34e257f0f573cfba6cd8e0c31c95cb46d"}
Oct 30 13:10:07.132671 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:07.132648    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf" event={"ID":"023728ea-7d4e-4d33-9618-320d46770bbd","Type":"ContainerStarted","Data":"7831d6e8cddad1676ab2ff44f104600fea9cb0880fb2b8b945fd3619fa0f1344"}
Oct 30 13:10:07.190379 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:07.190287    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf" podStartSLOduration=2.190254474 podStartE2EDuration="2.190254474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:10:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:10:07.185498015 +0000 UTC m=+1448.768927721" watchObservedRunningTime="2026-10-30 13:10:07.190254474 +0000 UTC m=+1448.773684166"
Oct 30 13:10:07.351155 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:07.351112    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:07.353029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:07.353004    5893 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:10:07.353176 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:07.353061    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf" podUID="023728ea-7d4e-4d33-9618-320d46770bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 13:10:07.833362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:07.833313    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt" podUID="7b6edbec-078c-41cc-8df9-d9f3643a5fad" containerName="payload-processing" containerID="cri-o://d675dd685f02772b525a61918b5e7ece7a674b700217cd20e7d900e2196a10cb" gracePeriod=25
Oct 30 13:10:07.874463 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:07.874363    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc" podUID="daa840f3-895e-492c-b87a-1369cf7f3504" containerName="payload-pre-processing" containerID="cri-o://aed17de4528e4482145b5857daa67c4f3154344e371b1e86b33ac32388aa14ae" gracePeriod=25
Oct 30 13:10:08.356100 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:08.355649    5893 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:10:08.356100 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:08.355708    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf" podUID="023728ea-7d4e-4d33-9618-320d46770bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 13:10:09.385757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:09.377420    5893 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:10:09.385757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:09.377489    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf" podUID="023728ea-7d4e-4d33-9618-320d46770bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 13:10:10.393345 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:10.392498    5893 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:10:10.393345 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:10.392565    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf" podUID="023728ea-7d4e-4d33-9618-320d46770bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 13:10:11.356299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:11.354681    5893 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:10:11.356299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:11.354743    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf" podUID="023728ea-7d4e-4d33-9618-320d46770bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 13:10:12.360300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:12.355518    5893 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:10:12.360300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:12.355578    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf" podUID="023728ea-7d4e-4d33-9618-320d46770bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 13:10:13.363300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:13.356638    5893 patch_prober.go:29] interesting pod/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:10:13.363300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:13.356705    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf" podUID="023728ea-7d4e-4d33-9618-320d46770bbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Oct 30 13:10:14.069409 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:14.069372    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt" podUID="7b6edbec-078c-41cc-8df9-d9f3643a5fad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:10:14.069586 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:14.069435    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc" podUID="daa840f3-895e-492c-b87a-1369cf7f3504" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:10:14.358091 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:14.357995    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:14.359108 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:14.359003    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:14.360162 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:14.360135    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:10:18.653823 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:10:18.653776    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:10:24.068405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:24.068363    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt" podUID="7b6edbec-078c-41cc-8df9-d9f3643a5fad" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:10:24.068405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:24.068398    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc" podUID="daa840f3-895e-492c-b87a-1369cf7f3504" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:10:28.189618 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.189589    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"
Oct 30 13:10:28.251153 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.251121    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b6edbec-078c-41cc-8df9-d9f3643a5fad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6edbec-078c-41cc-8df9-d9f3643a5fad-plugins-config-volume\") pod \"7b6edbec-078c-41cc-8df9-d9f3643a5fad\" (UID: \"7b6edbec-078c-41cc-8df9-d9f3643a5fad\") "
Oct 30 13:10:28.251361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.251310    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b6edbec-078c-41cc-8df9-d9f3643a5fad-kube-api-access-zblws\" (UniqueName: \"kubernetes.io/projected/7b6edbec-078c-41cc-8df9-d9f3643a5fad-kube-api-access-zblws\") pod \"7b6edbec-078c-41cc-8df9-d9f3643a5fad\" (UID: \"7b6edbec-078c-41cc-8df9-d9f3643a5fad\") "
Oct 30 13:10:28.251890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.251864    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b6edbec-078c-41cc-8df9-d9f3643a5fad-plugins-config-volume" pod "7b6edbec-078c-41cc-8df9-d9f3643a5fad" (UID: "7b6edbec-078c-41cc-8df9-d9f3643a5fad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:28.253702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.253680    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"
Oct 30 13:10:28.255043 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.254853    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b6edbec-078c-41cc-8df9-d9f3643a5fad-kube-api-access-zblws" pod "7b6edbec-078c-41cc-8df9-d9f3643a5fad" (UID: "7b6edbec-078c-41cc-8df9-d9f3643a5fad"). InnerVolumeSpecName "kube-api-access-zblws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:28.353441 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.353407    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daa840f3-895e-492c-b87a-1369cf7f3504-kube-api-access-gnfsv\" (UniqueName: \"kubernetes.io/projected/daa840f3-895e-492c-b87a-1369cf7f3504-kube-api-access-gnfsv\") pod \"daa840f3-895e-492c-b87a-1369cf7f3504\" (UID: \"daa840f3-895e-492c-b87a-1369cf7f3504\") "
Oct 30 13:10:28.353778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.353761    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daa840f3-895e-492c-b87a-1369cf7f3504-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa840f3-895e-492c-b87a-1369cf7f3504-plugins-config-volume\") pod \"daa840f3-895e-492c-b87a-1369cf7f3504\" (UID: \"daa840f3-895e-492c-b87a-1369cf7f3504\") "
Oct 30 13:10:28.354171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.354137    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa840f3-895e-492c-b87a-1369cf7f3504-plugins-config-volume" pod "daa840f3-895e-492c-b87a-1369cf7f3504" (UID: "daa840f3-895e-492c-b87a-1369cf7f3504"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:28.354595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.354570    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daa840f3-895e-492c-b87a-1369cf7f3504-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:10:28.354692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.354601    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b6edbec-078c-41cc-8df9-d9f3643a5fad-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:10:28.354692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.354615    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zblws\" (UniqueName: \"kubernetes.io/projected/7b6edbec-078c-41cc-8df9-d9f3643a5fad-kube-api-access-zblws\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:10:28.356062 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.356036    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa840f3-895e-492c-b87a-1369cf7f3504-kube-api-access-gnfsv" pod "daa840f3-895e-492c-b87a-1369cf7f3504" (UID: "daa840f3-895e-492c-b87a-1369cf7f3504"). InnerVolumeSpecName "kube-api-access-gnfsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:28.380418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.380344    5893 generic.go:356] "Generic (PLEG): container finished" podID="7b6edbec-078c-41cc-8df9-d9f3643a5fad" containerID="d675dd685f02772b525a61918b5e7ece7a674b700217cd20e7d900e2196a10cb" exitCode=0
Oct 30 13:10:28.380418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.380401    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"
Oct 30 13:10:28.380602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.380439    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt" event={"ID":"7b6edbec-078c-41cc-8df9-d9f3643a5fad","Type":"ContainerDied","Data":"d675dd685f02772b525a61918b5e7ece7a674b700217cd20e7d900e2196a10cb"}
Oct 30 13:10:28.380602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.380492    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt" event={"ID":"7b6edbec-078c-41cc-8df9-d9f3643a5fad","Type":"ContainerDied","Data":"21b157b5c71df5de2026723a7fbbf396bc65c1b5bf5ee9a57ef6d4b8bed298ea"}
Oct 30 13:10:28.380602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.380522    5893 scope.go:122] "RemoveContainer" containerID="d675dd685f02772b525a61918b5e7ece7a674b700217cd20e7d900e2196a10cb"
Oct 30 13:10:28.382100 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.382075    5893 generic.go:356] "Generic (PLEG): container finished" podID="daa840f3-895e-492c-b87a-1369cf7f3504" containerID="aed17de4528e4482145b5857daa67c4f3154344e371b1e86b33ac32388aa14ae" exitCode=0
Oct 30 13:10:28.382212 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.382131    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"
Oct 30 13:10:28.382212 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.382167    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc" event={"ID":"daa840f3-895e-492c-b87a-1369cf7f3504","Type":"ContainerDied","Data":"aed17de4528e4482145b5857daa67c4f3154344e371b1e86b33ac32388aa14ae"}
Oct 30 13:10:28.382518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.382219    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc" event={"ID":"daa840f3-895e-492c-b87a-1369cf7f3504","Type":"ContainerDied","Data":"35a476ae7ff99b85276bebeac0be8a3eb70a1687723fee98d6a6dc4d8ec2b53e"}
Oct 30 13:10:28.393063 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.393044    5893 scope.go:122] "RemoveContainer" containerID="d675dd685f02772b525a61918b5e7ece7a674b700217cd20e7d900e2196a10cb"
Oct 30 13:10:28.393434 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:10:28.393370    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d675dd685f02772b525a61918b5e7ece7a674b700217cd20e7d900e2196a10cb\": container with ID starting with d675dd685f02772b525a61918b5e7ece7a674b700217cd20e7d900e2196a10cb not found: ID does not exist" containerID="d675dd685f02772b525a61918b5e7ece7a674b700217cd20e7d900e2196a10cb"
Oct 30 13:10:28.393434 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.393405    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d675dd685f02772b525a61918b5e7ece7a674b700217cd20e7d900e2196a10cb"} err="failed to get container status \"d675dd685f02772b525a61918b5e7ece7a674b700217cd20e7d900e2196a10cb\": rpc error: code = NotFound desc = could not find container \"d675dd685f02772b525a61918b5e7ece7a674b700217cd20e7d900e2196a10cb\": container with ID starting with d675dd685f02772b525a61918b5e7ece7a674b700217cd20e7d900e2196a10cb not found: ID does not exist"
Oct 30 13:10:28.393434 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.393428    5893 scope.go:122] "RemoveContainer" containerID="aed17de4528e4482145b5857daa67c4f3154344e371b1e86b33ac32388aa14ae"
Oct 30 13:10:28.403990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.403972    5893 scope.go:122] "RemoveContainer" containerID="aed17de4528e4482145b5857daa67c4f3154344e371b1e86b33ac32388aa14ae"
Oct 30 13:10:28.404264 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:10:28.404243    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed17de4528e4482145b5857daa67c4f3154344e371b1e86b33ac32388aa14ae\": container with ID starting with aed17de4528e4482145b5857daa67c4f3154344e371b1e86b33ac32388aa14ae not found: ID does not exist" containerID="aed17de4528e4482145b5857daa67c4f3154344e371b1e86b33ac32388aa14ae"
Oct 30 13:10:28.404371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.404287    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed17de4528e4482145b5857daa67c4f3154344e371b1e86b33ac32388aa14ae"} err="failed to get container status \"aed17de4528e4482145b5857daa67c4f3154344e371b1e86b33ac32388aa14ae\": rpc error: code = NotFound desc = could not find container \"aed17de4528e4482145b5857daa67c4f3154344e371b1e86b33ac32388aa14ae\": container with ID starting with aed17de4528e4482145b5857daa67c4f3154344e371b1e86b33ac32388aa14ae not found: ID does not exist"
Oct 30 13:10:28.433386 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.433357    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"]
Oct 30 13:10:28.437365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.437340    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-6g6nc"]
Oct 30 13:10:28.456810 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.456778    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnfsv\" (UniqueName: \"kubernetes.io/projected/daa840f3-895e-492c-b87a-1369cf7f3504-kube-api-access-gnfsv\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:10:28.473038 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.473008    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"]
Oct 30 13:10:28.484155 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.484063    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-knlmt"]
Oct 30 13:10:28.532717 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.532685    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b6edbec-078c-41cc-8df9-d9f3643a5fad" path="/var/lib/kubelet/pods/7b6edbec-078c-41cc-8df9-d9f3643a5fad/volumes"
Oct 30 13:10:28.533145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:28.533132    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daa840f3-895e-492c-b87a-1369cf7f3504" path="/var/lib/kubelet/pods/daa840f3-895e-492c-b87a-1369cf7f3504/volumes"
Oct 30 13:10:42.018468 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:42.018433    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"]
Oct 30 13:10:42.018982 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:42.018631    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m" podUID="af82e4b8-5b4c-4505-a2f4-ee22f28cade0" containerName="istio-proxy" containerID="cri-o://ecd896b350da7c063d420cc845da81f82fdb0e1ed18d08a3e2ebbf5a8b8ad928" gracePeriod=30
Oct 30 13:10:47.358336 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.358309    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:10:47.434882 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.434850    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-workload-socket\") pod \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") "
Oct 30 13:10:47.435062 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.434924    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-token\" (UniqueName: \"kubernetes.io/projected/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-token\") pod \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") "
Oct 30 13:10:47.435062 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.434963    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istiod-ca-cert\") pod \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") "
Oct 30 13:10:47.435062 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.435044    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-credential-socket\") pod \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") "
Oct 30 13:10:47.435236 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.435134    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-data\") pod \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") "
Oct 30 13:10:47.435421 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.435389    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-podinfo\") pod \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") "
Oct 30 13:10:47.435503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.435460    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-kube-api-access-czh9q\" (UniqueName: \"kubernetes.io/projected/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-kube-api-access-czh9q\") pod \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") "
Oct 30 13:10:47.435503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.435494    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-workload-certs\") pod \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") "
Oct 30 13:10:47.435603 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.435528    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-envoy\") pod \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\" (UID: \"af82e4b8-5b4c-4505-a2f4-ee22f28cade0\") "
Oct 30 13:10:47.437077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.436735    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-workload-socket" pod "af82e4b8-5b4c-4505-a2f4-ee22f28cade0" (UID: "af82e4b8-5b4c-4505-a2f4-ee22f28cade0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:47.437077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.436969    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-data" pod "af82e4b8-5b4c-4505-a2f4-ee22f28cade0" (UID: "af82e4b8-5b4c-4505-a2f4-ee22f28cade0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:47.437332 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.436232    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-credential-socket" pod "af82e4b8-5b4c-4505-a2f4-ee22f28cade0" (UID: "af82e4b8-5b4c-4505-a2f4-ee22f28cade0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:47.437332 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.437163    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-workload-certs" pod "af82e4b8-5b4c-4505-a2f4-ee22f28cade0" (UID: "af82e4b8-5b4c-4505-a2f4-ee22f28cade0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:47.437911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.437887    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:10:47.438005 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.437917    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:10:47.438005 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.437931    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:10:47.438005 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.437946    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:10:47.438213 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.438190    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istiod-ca-cert" pod "af82e4b8-5b4c-4505-a2f4-ee22f28cade0" (UID: "af82e4b8-5b4c-4505-a2f4-ee22f28cade0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:10:47.458506 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.458473    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-kube-api-access-czh9q" pod "af82e4b8-5b4c-4505-a2f4-ee22f28cade0" (UID: "af82e4b8-5b4c-4505-a2f4-ee22f28cade0"). InnerVolumeSpecName "kube-api-access-czh9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:47.465165 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.465118    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-envoy" pod "af82e4b8-5b4c-4505-a2f4-ee22f28cade0" (UID: "af82e4b8-5b4c-4505-a2f4-ee22f28cade0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:10:47.465303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.465160    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-token" pod "af82e4b8-5b4c-4505-a2f4-ee22f28cade0" (UID: "af82e4b8-5b4c-4505-a2f4-ee22f28cade0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:47.466948 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.466910    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-podinfo" pod "af82e4b8-5b4c-4505-a2f4-ee22f28cade0" (UID: "af82e4b8-5b4c-4505-a2f4-ee22f28cade0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:10:47.539733 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.539604    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:10:47.539733 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.539640    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czh9q\" (UniqueName: \"kubernetes.io/projected/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-kube-api-access-czh9q\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:10:47.539733 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.539654    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:10:47.539733 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.539667    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:10:47.539733 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.539682    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af82e4b8-5b4c-4505-a2f4-ee22f28cade0-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:10:47.779723 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.779673    5893 generic.go:356] "Generic (PLEG): container finished" podID="af82e4b8-5b4c-4505-a2f4-ee22f28cade0" containerID="ecd896b350da7c063d420cc845da81f82fdb0e1ed18d08a3e2ebbf5a8b8ad928" exitCode=0
Oct 30 13:10:47.780420 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.779803    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"
Oct 30 13:10:47.785897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.779822    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m" event={"ID":"af82e4b8-5b4c-4505-a2f4-ee22f28cade0","Type":"ContainerDied","Data":"ecd896b350da7c063d420cc845da81f82fdb0e1ed18d08a3e2ebbf5a8b8ad928"}
Oct 30 13:10:47.786020 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.785971    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m" event={"ID":"af82e4b8-5b4c-4505-a2f4-ee22f28cade0","Type":"ContainerDied","Data":"8b4978f9a7cfabde3809d3d1e9523ecb597f7d7a1cb5b472838cd875df86feae"}
Oct 30 13:10:47.786020 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.786016    5893 scope.go:122] "RemoveContainer" containerID="ecd896b350da7c063d420cc845da81f82fdb0e1ed18d08a3e2ebbf5a8b8ad928"
Oct 30 13:10:47.812984 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.812954    5893 scope.go:122] "RemoveContainer" containerID="ecd896b350da7c063d420cc845da81f82fdb0e1ed18d08a3e2ebbf5a8b8ad928"
Oct 30 13:10:47.813355 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:10:47.813331    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecd896b350da7c063d420cc845da81f82fdb0e1ed18d08a3e2ebbf5a8b8ad928\": container with ID starting with ecd896b350da7c063d420cc845da81f82fdb0e1ed18d08a3e2ebbf5a8b8ad928 not found: ID does not exist" containerID="ecd896b350da7c063d420cc845da81f82fdb0e1ed18d08a3e2ebbf5a8b8ad928"
Oct 30 13:10:47.813455 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.813368    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecd896b350da7c063d420cc845da81f82fdb0e1ed18d08a3e2ebbf5a8b8ad928"} err="failed to get container status \"ecd896b350da7c063d420cc845da81f82fdb0e1ed18d08a3e2ebbf5a8b8ad928\": rpc error: code = NotFound desc = could not find container \"ecd896b350da7c063d420cc845da81f82fdb0e1ed18d08a3e2ebbf5a8b8ad928\": container with ID starting with ecd896b350da7c063d420cc845da81f82fdb0e1ed18d08a3e2ebbf5a8b8ad928 not found: ID does not exist"
Oct 30 13:10:47.849614 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.849583    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"]
Oct 30 13:10:47.864073 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:47.864041    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-sw66m"]
Oct 30 13:10:48.533029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:10:48.532998    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af82e4b8-5b4c-4505-a2f4-ee22f28cade0" path="/var/lib/kubelet/pods/af82e4b8-5b4c-4505-a2f4-ee22f28cade0/volumes"
Oct 30 13:10:48.649535 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:10:48.649493    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:11:00.908992 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:00.908958    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:11:00.910424 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:00.910399    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:11:00.924000 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:00.923974    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:11:00.924177 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:00.923985    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:11:09.341310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:09.341231    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"]
Oct 30 13:11:14.366304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:14.366231    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c" podUID="d1a54829-a667-49b5-9f16-f8ae8b10218e" containerName="payload-pre-processing" containerID="cri-o://935d8159e2d174dc81c608273d50b5c8f368cb559d762195e55cfc12de02c27c" gracePeriod=25
Oct 30 13:11:14.714569 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:14.714539    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"
Oct 30 13:11:14.814323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:14.814263    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a54829-a667-49b5-9f16-f8ae8b10218e-kube-api-access-kzfh8\" (UniqueName: \"kubernetes.io/projected/d1a54829-a667-49b5-9f16-f8ae8b10218e-kube-api-access-kzfh8\") pod \"d1a54829-a667-49b5-9f16-f8ae8b10218e\" (UID: \"d1a54829-a667-49b5-9f16-f8ae8b10218e\") "
Oct 30 13:11:14.814518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:14.814360    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a54829-a667-49b5-9f16-f8ae8b10218e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a54829-a667-49b5-9f16-f8ae8b10218e-plugins-config-volume\") pod \"d1a54829-a667-49b5-9f16-f8ae8b10218e\" (UID: \"d1a54829-a667-49b5-9f16-f8ae8b10218e\") "
Oct 30 13:11:14.815527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:14.815503    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a54829-a667-49b5-9f16-f8ae8b10218e-plugins-config-volume" pod "d1a54829-a667-49b5-9f16-f8ae8b10218e" (UID: "d1a54829-a667-49b5-9f16-f8ae8b10218e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:14.817815 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:14.817779    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a54829-a667-49b5-9f16-f8ae8b10218e-kube-api-access-kzfh8" pod "d1a54829-a667-49b5-9f16-f8ae8b10218e" (UID: "d1a54829-a667-49b5-9f16-f8ae8b10218e"). InnerVolumeSpecName "kube-api-access-kzfh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:14.917107 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:14.917077    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzfh8\" (UniqueName: \"kubernetes.io/projected/d1a54829-a667-49b5-9f16-f8ae8b10218e-kube-api-access-kzfh8\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:14.917107 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:14.917102    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1a54829-a667-49b5-9f16-f8ae8b10218e-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:15.243237 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:15.243198    5893 generic.go:356] "Generic (PLEG): container finished" podID="d1a54829-a667-49b5-9f16-f8ae8b10218e" containerID="935d8159e2d174dc81c608273d50b5c8f368cb559d762195e55cfc12de02c27c" exitCode=0
Oct 30 13:11:15.243414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:15.243244    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"
Oct 30 13:11:15.243414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:15.243308    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c" event={"ID":"d1a54829-a667-49b5-9f16-f8ae8b10218e","Type":"ContainerDied","Data":"935d8159e2d174dc81c608273d50b5c8f368cb559d762195e55cfc12de02c27c"}
Oct 30 13:11:15.243414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:15.243372    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c" event={"ID":"d1a54829-a667-49b5-9f16-f8ae8b10218e","Type":"ContainerDied","Data":"d298fbb07340a2b27467cf9e6c2820817084b489a7a311d0b23f11f8e14b211b"}
Oct 30 13:11:15.243414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:15.243396    5893 scope.go:122] "RemoveContainer" containerID="935d8159e2d174dc81c608273d50b5c8f368cb559d762195e55cfc12de02c27c"
Oct 30 13:11:15.259351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:15.259326    5893 scope.go:122] "RemoveContainer" containerID="935d8159e2d174dc81c608273d50b5c8f368cb559d762195e55cfc12de02c27c"
Oct 30 13:11:15.259729 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:11:15.259710    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935d8159e2d174dc81c608273d50b5c8f368cb559d762195e55cfc12de02c27c\": container with ID starting with 935d8159e2d174dc81c608273d50b5c8f368cb559d762195e55cfc12de02c27c not found: ID does not exist" containerID="935d8159e2d174dc81c608273d50b5c8f368cb559d762195e55cfc12de02c27c"
Oct 30 13:11:15.259799 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:15.259740    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935d8159e2d174dc81c608273d50b5c8f368cb559d762195e55cfc12de02c27c"} err="failed to get container status \"935d8159e2d174dc81c608273d50b5c8f368cb559d762195e55cfc12de02c27c\": rpc error: code = NotFound desc = could not find container \"935d8159e2d174dc81c608273d50b5c8f368cb559d762195e55cfc12de02c27c\": container with ID starting with 935d8159e2d174dc81c608273d50b5c8f368cb559d762195e55cfc12de02c27c not found: ID does not exist"
Oct 30 13:11:15.297533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:15.297506    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"]
Oct 30 13:11:15.300899 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:15.300873    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-17fc433e-58fcb84879-rzd8c"]
Oct 30 13:11:16.534729 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:16.534694    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a54829-a667-49b5-9f16-f8ae8b10218e" path="/var/lib/kubelet/pods/d1a54829-a667-49b5-9f16-f8ae8b10218e/volumes"
Oct 30 13:11:18.070487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.070452    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"]
Oct 30 13:11:18.071645 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.071588    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk" podUID="8d55cc4a-7c7b-4171-82f6-f274324b2228" containerName="maas-api" containerID="cri-o://cbd41e42df51561d6c4cf5759a7d6a74bd0d39fce549d9c8d2a7869c84bb2274" gracePeriod=30
Oct 30 13:11:18.284531 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.284489    5893 generic.go:356] "Generic (PLEG): container finished" podID="8d55cc4a-7c7b-4171-82f6-f274324b2228" containerID="cbd41e42df51561d6c4cf5759a7d6a74bd0d39fce549d9c8d2a7869c84bb2274" exitCode=0
Oct 30 13:11:18.284779 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.284757    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk" event={"ID":"8d55cc4a-7c7b-4171-82f6-f274324b2228","Type":"ContainerDied","Data":"cbd41e42df51561d6c4cf5759a7d6a74bd0d39fce549d9c8d2a7869c84bb2274"}
Oct 30 13:11:18.421860 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.421828    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:11:18.572679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.564436    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d55cc4a-7c7b-4171-82f6-f274324b2228-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d55cc4a-7c7b-4171-82f6-f274324b2228-metrics-certs\") pod \"8d55cc4a-7c7b-4171-82f6-f274324b2228\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") "
Oct 30 13:11:18.572679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.564552    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d55cc4a-7c7b-4171-82f6-f274324b2228-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d55cc4a-7c7b-4171-82f6-f274324b2228-openshift-service-ca\") pod \"8d55cc4a-7c7b-4171-82f6-f274324b2228\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") "
Oct 30 13:11:18.572679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.564673    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d55cc4a-7c7b-4171-82f6-f274324b2228-kube-api-access-6q9cr\" (UniqueName: \"kubernetes.io/projected/8d55cc4a-7c7b-4171-82f6-f274324b2228-kube-api-access-6q9cr\") pod \"8d55cc4a-7c7b-4171-82f6-f274324b2228\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") "
Oct 30 13:11:18.572679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.564713    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d55cc4a-7c7b-4171-82f6-f274324b2228-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d55cc4a-7c7b-4171-82f6-f274324b2228-maas-api-tls\") pod \"8d55cc4a-7c7b-4171-82f6-f274324b2228\" (UID: \"8d55cc4a-7c7b-4171-82f6-f274324b2228\") "
Oct 30 13:11:18.572679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.567451    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d55cc4a-7c7b-4171-82f6-f274324b2228-openshift-service-ca" pod "8d55cc4a-7c7b-4171-82f6-f274324b2228" (UID: "8d55cc4a-7c7b-4171-82f6-f274324b2228"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:18.572679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.569886    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d55cc4a-7c7b-4171-82f6-f274324b2228-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:18.598324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.597373    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d55cc4a-7c7b-4171-82f6-f274324b2228-metrics-certs" pod "8d55cc4a-7c7b-4171-82f6-f274324b2228" (UID: "8d55cc4a-7c7b-4171-82f6-f274324b2228"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:18.613975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.611519    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d55cc4a-7c7b-4171-82f6-f274324b2228-kube-api-access-6q9cr" pod "8d55cc4a-7c7b-4171-82f6-f274324b2228" (UID: "8d55cc4a-7c7b-4171-82f6-f274324b2228"). InnerVolumeSpecName "kube-api-access-6q9cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:18.613975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.612958    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d55cc4a-7c7b-4171-82f6-f274324b2228-maas-api-tls" pod "8d55cc4a-7c7b-4171-82f6-f274324b2228" (UID: "8d55cc4a-7c7b-4171-82f6-f274324b2228"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:18.679386 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.677240    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q9cr\" (UniqueName: \"kubernetes.io/projected/8d55cc4a-7c7b-4171-82f6-f274324b2228-kube-api-access-6q9cr\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:18.679386 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.677297    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8d55cc4a-7c7b-4171-82f6-f274324b2228-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:18.679386 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:18.677313    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d55cc4a-7c7b-4171-82f6-f274324b2228-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:18.714004 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:11:18.713847    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:11:19.300669 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:19.300639    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"
Oct 30 13:11:19.301200 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:19.300689    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk" event={"ID":"8d55cc4a-7c7b-4171-82f6-f274324b2228","Type":"ContainerDied","Data":"dffe7f97f43fe8c95de243f7656ae511e4ace6576408a856cf5ba275e60dd08f"}
Oct 30 13:11:19.301200 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:19.300754    5893 scope.go:122] "RemoveContainer" containerID="cbd41e42df51561d6c4cf5759a7d6a74bd0d39fce549d9c8d2a7869c84bb2274"
Oct 30 13:11:19.359771 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:19.359737    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"]
Oct 30 13:11:19.370383 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:19.370339    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-17fc433e-7d69658bff-4gbfk"]
Oct 30 13:11:20.532767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:20.532739    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d55cc4a-7c7b-4171-82f6-f274324b2228" path="/var/lib/kubelet/pods/8d55cc4a-7c7b-4171-82f6-f274324b2228/volumes"
Oct 30 13:11:26.919978 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:26.892172    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"]
Oct 30 13:11:26.919978 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:26.892473    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" podUID="e200c2e3-c88e-4707-a139-c5b8dfab3941" containerName="istio-proxy" containerID="cri-o://8fae83a6c49bbbacfb3e2d63963639842ed9819299fe06fa2f163581b6b87bd4" gracePeriod=30
Oct 30 13:11:26.959505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:26.959467    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"]
Oct 30 13:11:26.960991 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:26.959725    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n" podUID="a39cee14-750a-47d1-8b5c-d362c150a6d1" containerName="istio-proxy" containerID="cri-o://e56c7ffefc4fb93b75b2112f3a71f0604204fb7097d50c467780e1e79b0bfdf3" gracePeriod=30
Oct 30 13:11:26.989554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:26.989336    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"]
Oct 30 13:11:26.989747 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:26.989699    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64" podUID="534048b5-3bb5-4eab-9357-9a464373036b" containerName="istio-proxy" containerID="cri-o://c5aecc44f4e5e95ec87e8fda8655e05a6dc5aa6ff034b5a786a93d0f283057d6" gracePeriod=30
Oct 30 13:11:26.993449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:26.992966    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"]
Oct 30 13:11:26.993449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:26.993187    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp" podUID="42abda8f-3710-4edc-8bbb-f032aa9fa4f6" containerName="istio-proxy" containerID="cri-o://7da42dca14ce2b47c076bb91da94c1b5622306cf0a0072dc5047969d268ea1a7" gracePeriod=30
Oct 30 13:11:26.996665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:26.996618    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"]
Oct 30 13:11:26.997084 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:26.996874    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7" podUID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerName="istio-proxy" containerID="cri-o://a022a76d332d34826321a5583f4fc8dd29b13e1df2b48af9ce86a9990dec8a0d" gracePeriod=30
Oct 30 13:11:29.278029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.277992    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"]
Oct 30 13:11:29.279216 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279190    5893 cpu_manager.go:414] "containerMap: removing container" podUID="7b6edbec-078c-41cc-8df9-d9f3643a5fad" containerName="payload-processing"
Oct 30 13:11:29.279216 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279214    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b6edbec-078c-41cc-8df9-d9f3643a5fad" containerName="payload-processing"
Oct 30 13:11:29.279469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279235    5893 cpu_manager.go:414] "containerMap: removing container" podUID="af82e4b8-5b4c-4505-a2f4-ee22f28cade0" containerName="istio-proxy"
Oct 30 13:11:29.279469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279244    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af82e4b8-5b4c-4505-a2f4-ee22f28cade0" containerName="istio-proxy"
Oct 30 13:11:29.279469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279340    5893 cpu_manager.go:414] "containerMap: removing container" podUID="daa840f3-895e-492c-b87a-1369cf7f3504" containerName="payload-pre-processing"
Oct 30 13:11:29.279469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279352    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daa840f3-895e-492c-b87a-1369cf7f3504" containerName="payload-pre-processing"
Oct 30 13:11:29.279469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279372    5893 cpu_manager.go:414] "containerMap: removing container" podUID="8d55cc4a-7c7b-4171-82f6-f274324b2228" containerName="maas-api"
Oct 30 13:11:29.279469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279379    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d55cc4a-7c7b-4171-82f6-f274324b2228" containerName="maas-api"
Oct 30 13:11:29.279469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279449    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d1a54829-a667-49b5-9f16-f8ae8b10218e" containerName="payload-pre-processing"
Oct 30 13:11:29.279469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279456    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a54829-a667-49b5-9f16-f8ae8b10218e" containerName="payload-pre-processing"
Oct 30 13:11:29.280440 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279674    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="af82e4b8-5b4c-4505-a2f4-ee22f28cade0" containerName="istio-proxy"
Oct 30 13:11:29.280440 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279693    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="daa840f3-895e-492c-b87a-1369cf7f3504" containerName="payload-pre-processing"
Oct 30 13:11:29.280440 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279704    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d55cc4a-7c7b-4171-82f6-f274324b2228" containerName="maas-api"
Oct 30 13:11:29.280440 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279725    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a54829-a667-49b5-9f16-f8ae8b10218e" containerName="payload-pre-processing"
Oct 30 13:11:29.280440 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.279739    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b6edbec-078c-41cc-8df9-d9f3643a5fad" containerName="payload-processing"
Oct 30 13:11:29.285804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.285782    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.303755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.303730    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-abbc6c-openshift-default-dockercfg-qkmsh\""
Oct 30 13:11:29.305740 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.305718    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"]
Oct 30 13:11:29.407445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.407406    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-data\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.407614 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.407482    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-token\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.407708 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.407651    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-envoy\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.407708 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.407693    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-workload-socket\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.407813 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.407755    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-podinfo\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.407813 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.407794    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-credential-socket\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.407970 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.407849    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-workload-certs\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.407970 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.407885    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8kj\" (UniqueName: \"kubernetes.io/projected/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-kube-api-access-lt8kj\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.408103 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.408025    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istiod-ca-cert\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.510300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.509422    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-data\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.510300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.509498    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-token\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.510300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.509571    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-envoy\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.510300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.509598    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-workload-socket\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.510300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.509647    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-podinfo\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.510300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.509677    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-credential-socket\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.510300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.509711    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-workload-certs\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.510300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.509742    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8kj\" (UniqueName: \"kubernetes.io/projected/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-kube-api-access-lt8kj\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.510300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.509859    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istiod-ca-cert\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.510985 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.510763    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istiod-ca-cert\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.514613 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.511051    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-data\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.538347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.526725    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-credential-socket\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.538347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.526981    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-workload-socket\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.626296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.577714    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-workload-certs\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.626296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.588256    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-token\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.626296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.623210    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-podinfo\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.626296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.623562    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-envoy\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.626666 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.626368    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8kj\" (UniqueName: \"kubernetes.io/projected/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-kube-api-access-lt8kj\") pod \"e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") " pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:29.915190 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:29.915021    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:30.260090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:30.260049    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"]
Oct 30 13:11:30.262507 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:11:30.262476    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed508c1_ee5b_47e1_b335_7453b53a6f9d.slice/crio-600739324be10af9e9252f1932dea815d25459f0897a2d42efe2a40f81fba523 WatchSource:0}: Error finding container 600739324be10af9e9252f1932dea815d25459f0897a2d42efe2a40f81fba523: Status 404 returned error can't find the container with id 600739324be10af9e9252f1932dea815d25459f0897a2d42efe2a40f81fba523
Oct 30 13:11:30.266544 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:30.266502    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:30.266633 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:30.266598    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:30.266672 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:30.266643    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:30.420432 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:30.420388    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" event={"ID":"2ed508c1-ee5b-47e1-b335-7453b53a6f9d","Type":"ContainerStarted","Data":"0074433727b7e1099de818eb07ed8ab63cce1d10de7fdb2641bc9fc90a1b1b70"}
Oct 30 13:11:30.420912 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:30.420462    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" event={"ID":"2ed508c1-ee5b-47e1-b335-7453b53a6f9d","Type":"ContainerStarted","Data":"600739324be10af9e9252f1932dea815d25459f0897a2d42efe2a40f81fba523"}
Oct 30 13:11:30.475785 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:30.475736    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" podStartSLOduration=1.475722991 podStartE2EDuration="1.475722991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:30.472370396 +0000 UTC m=+1532.055800090" watchObservedRunningTime="2026-10-30 13:11:30.475722991 +0000 UTC m=+1532.059152683"
Oct 30 13:11:30.915402 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:30.915369    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:31.934842 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:31.934814    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:32.353482 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.352459    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:11:32.374247 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.373756    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:11:32.446652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.445496    5893 generic.go:356] "Generic (PLEG): container finished" podID="a39cee14-750a-47d1-8b5c-d362c150a6d1" containerID="e56c7ffefc4fb93b75b2112f3a71f0604204fb7097d50c467780e1e79b0bfdf3" exitCode=0
Oct 30 13:11:32.446807 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.446661    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n" event={"ID":"a39cee14-750a-47d1-8b5c-d362c150a6d1","Type":"ContainerDied","Data":"e56c7ffefc4fb93b75b2112f3a71f0604204fb7097d50c467780e1e79b0bfdf3"}
Oct 30 13:11:32.454344 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.452585    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:11:32.462906 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.462876    5893 generic.go:356] "Generic (PLEG): container finished" podID="534048b5-3bb5-4eab-9357-9a464373036b" containerID="c5aecc44f4e5e95ec87e8fda8655e05a6dc5aa6ff034b5a786a93d0f283057d6" exitCode=0
Oct 30 13:11:32.463211 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.463189    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64" event={"ID":"534048b5-3bb5-4eab-9357-9a464373036b","Type":"ContainerDied","Data":"c5aecc44f4e5e95ec87e8fda8655e05a6dc5aa6ff034b5a786a93d0f283057d6"}
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.470660    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-workload-socket\") pod \"e200c2e3-c88e-4707-a139-c5b8dfab3941\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.470748    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-data\") pod \"e200c2e3-c88e-4707-a139-c5b8dfab3941\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.470857    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-kube-api-access-58cq6\" (UniqueName: \"kubernetes.io/projected/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-kube-api-access-58cq6\") pod \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.470894    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-credential-socket\") pod \"e200c2e3-c88e-4707-a139-c5b8dfab3941\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.470933    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-workload-certs\") pod \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.470975    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-workload-certs\") pod \"e200c2e3-c88e-4707-a139-c5b8dfab3941\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.471022    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-token\" (UniqueName: \"kubernetes.io/projected/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-token\") pod \"e200c2e3-c88e-4707-a139-c5b8dfab3941\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.471100    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-envoy\") pod \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.471170    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istiod-ca-cert\") pod \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.471225    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e200c2e3-c88e-4707-a139-c5b8dfab3941-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e200c2e3-c88e-4707-a139-c5b8dfab3941-istiod-ca-cert\") pod \"e200c2e3-c88e-4707-a139-c5b8dfab3941\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.471288    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-token\" (UniqueName: \"kubernetes.io/projected/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-token\") pod \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.471363    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-envoy\") pod \"e200c2e3-c88e-4707-a139-c5b8dfab3941\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.471390    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-data\") pod \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.471436    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-podinfo\") pod \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.471515    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-workload-socket\") pod \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.471545    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e200c2e3-c88e-4707-a139-c5b8dfab3941-kube-api-access-bwdd4\" (UniqueName: \"kubernetes.io/projected/e200c2e3-c88e-4707-a139-c5b8dfab3941-kube-api-access-bwdd4\") pod \"e200c2e3-c88e-4707-a139-c5b8dfab3941\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.471595    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-credential-socket\") pod \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\" (UID: \"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7\") "
Oct 30 13:11:32.473507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.471615    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-podinfo\") pod \"e200c2e3-c88e-4707-a139-c5b8dfab3941\" (UID: \"e200c2e3-c88e-4707-a139-c5b8dfab3941\") "
Oct 30 13:11:32.475250 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.474976    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-credential-socket" pod "e200c2e3-c88e-4707-a139-c5b8dfab3941" (UID: "e200c2e3-c88e-4707-a139-c5b8dfab3941"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.475250 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.475151    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-workload-socket" pod "e200c2e3-c88e-4707-a139-c5b8dfab3941" (UID: "e200c2e3-c88e-4707-a139-c5b8dfab3941"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.475370 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.475345    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-data" pod "e200c2e3-c88e-4707-a139-c5b8dfab3941" (UID: "e200c2e3-c88e-4707-a139-c5b8dfab3941"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.475556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.475527    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-podinfo" pod "e200c2e3-c88e-4707-a139-c5b8dfab3941" (UID: "e200c2e3-c88e-4707-a139-c5b8dfab3941"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:11:32.475993 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.475967    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e200c2e3-c88e-4707-a139-c5b8dfab3941-istiod-ca-cert" pod "e200c2e3-c88e-4707-a139-c5b8dfab3941" (UID: "e200c2e3-c88e-4707-a139-c5b8dfab3941"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:32.477566 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.477536    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-kube-api-access-58cq6" pod "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" (UID: "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7"). InnerVolumeSpecName "kube-api-access-58cq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:32.477977 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.477953    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-workload-certs" pod "e200c2e3-c88e-4707-a139-c5b8dfab3941" (UID: "e200c2e3-c88e-4707-a139-c5b8dfab3941"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.478180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.478156    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-workload-certs" pod "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" (UID: "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.480726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.480519    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-podinfo" pod "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" (UID: "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:11:32.481064 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.481009    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-token" pod "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" (UID: "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:32.481343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.481223    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-credential-socket" pod "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" (UID: "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.483337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.483122    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e200c2e3-c88e-4707-a139-c5b8dfab3941-kube-api-access-bwdd4" pod "e200c2e3-c88e-4707-a139-c5b8dfab3941" (UID: "e200c2e3-c88e-4707-a139-c5b8dfab3941"). InnerVolumeSpecName "kube-api-access-bwdd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:32.484086 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.483476    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-workload-socket" pod "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" (UID: "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.484201 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.484171    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-envoy" pod "e200c2e3-c88e-4707-a139-c5b8dfab3941" (UID: "e200c2e3-c88e-4707-a139-c5b8dfab3941"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.484421 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.484339    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-data" pod "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" (UID: "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.485868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.485433    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-token" pod "e200c2e3-c88e-4707-a139-c5b8dfab3941" (UID: "e200c2e3-c88e-4707-a139-c5b8dfab3941"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:32.488690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.488665    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istiod-ca-cert" pod "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" (UID: "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:32.490567 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.490507    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-envoy" pod "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" (UID: "f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.490653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.490603    5893 generic.go:356] "Generic (PLEG): container finished" podID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerID="a022a76d332d34826321a5583f4fc8dd29b13e1df2b48af9ce86a9990dec8a0d" exitCode=0
Oct 30 13:11:32.490714 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.490681    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"
Oct 30 13:11:32.490768 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.490724    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7" event={"ID":"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7","Type":"ContainerDied","Data":"a022a76d332d34826321a5583f4fc8dd29b13e1df2b48af9ce86a9990dec8a0d"}
Oct 30 13:11:32.491212 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.491183    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7" event={"ID":"f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7","Type":"ContainerDied","Data":"654fdac0024f4fc233d3f40ff360ff2c97685db1434f41fce140b5c8e17c3429"}
Oct 30 13:11:32.491467 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.491245    5893 scope.go:122] "RemoveContainer" containerID="a022a76d332d34826321a5583f4fc8dd29b13e1df2b48af9ce86a9990dec8a0d"
Oct 30 13:11:32.494754 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.494718    5893 generic.go:356] "Generic (PLEG): container finished" podID="42abda8f-3710-4edc-8bbb-f032aa9fa4f6" containerID="7da42dca14ce2b47c076bb91da94c1b5622306cf0a0072dc5047969d268ea1a7" exitCode=0
Oct 30 13:11:32.494866 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.494815    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"
Oct 30 13:11:32.494942 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.494852    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp" event={"ID":"42abda8f-3710-4edc-8bbb-f032aa9fa4f6","Type":"ContainerDied","Data":"7da42dca14ce2b47c076bb91da94c1b5622306cf0a0072dc5047969d268ea1a7"}
Oct 30 13:11:32.499995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.499967    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:11:32.504875 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.504851    5893 generic.go:356] "Generic (PLEG): container finished" podID="e200c2e3-c88e-4707-a139-c5b8dfab3941" containerID="8fae83a6c49bbbacfb3e2d63963639842ed9819299fe06fa2f163581b6b87bd4" exitCode=0
Oct 30 13:11:32.505061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.504933    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"
Oct 30 13:11:32.505061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.504941    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" event={"ID":"e200c2e3-c88e-4707-a139-c5b8dfab3941","Type":"ContainerDied","Data":"8fae83a6c49bbbacfb3e2d63963639842ed9819299fe06fa2f163581b6b87bd4"}
Oct 30 13:11:32.505061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.505009    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb" event={"ID":"e200c2e3-c88e-4707-a139-c5b8dfab3941","Type":"ContainerDied","Data":"6a8496c905999d3f2b12c1ffbffa6a58e9cb5259e910a0be5fae86e6bc23865c"}
Oct 30 13:11:32.505424 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.505407    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:32.517113 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.510059    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:11:32.517113 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.514767    5893 scope.go:122] "RemoveContainer" containerID="0fc25ddcb9028983f0fa28caede55b2289399949b584b34089f7e4f75e2f1763"
Oct 30 13:11:32.550531 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.548675    5893 scope.go:122] "RemoveContainer" containerID="a022a76d332d34826321a5583f4fc8dd29b13e1df2b48af9ce86a9990dec8a0d"
Oct 30 13:11:32.550531 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:11:32.549151    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a022a76d332d34826321a5583f4fc8dd29b13e1df2b48af9ce86a9990dec8a0d\": container with ID starting with a022a76d332d34826321a5583f4fc8dd29b13e1df2b48af9ce86a9990dec8a0d not found: ID does not exist" containerID="a022a76d332d34826321a5583f4fc8dd29b13e1df2b48af9ce86a9990dec8a0d"
Oct 30 13:11:32.550531 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.549193    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a022a76d332d34826321a5583f4fc8dd29b13e1df2b48af9ce86a9990dec8a0d"} err="failed to get container status \"a022a76d332d34826321a5583f4fc8dd29b13e1df2b48af9ce86a9990dec8a0d\": rpc error: code = NotFound desc = could not find container \"a022a76d332d34826321a5583f4fc8dd29b13e1df2b48af9ce86a9990dec8a0d\": container with ID starting with a022a76d332d34826321a5583f4fc8dd29b13e1df2b48af9ce86a9990dec8a0d not found: ID does not exist"
Oct 30 13:11:32.550531 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.549218    5893 scope.go:122] "RemoveContainer" containerID="0fc25ddcb9028983f0fa28caede55b2289399949b584b34089f7e4f75e2f1763"
Oct 30 13:11:32.550531 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:11:32.549656    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc25ddcb9028983f0fa28caede55b2289399949b584b34089f7e4f75e2f1763\": container with ID starting with 0fc25ddcb9028983f0fa28caede55b2289399949b584b34089f7e4f75e2f1763 not found: ID does not exist" containerID="0fc25ddcb9028983f0fa28caede55b2289399949b584b34089f7e4f75e2f1763"
Oct 30 13:11:32.550531 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.549710    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc25ddcb9028983f0fa28caede55b2289399949b584b34089f7e4f75e2f1763"} err="failed to get container status \"0fc25ddcb9028983f0fa28caede55b2289399949b584b34089f7e4f75e2f1763\": rpc error: code = NotFound desc = could not find container \"0fc25ddcb9028983f0fa28caede55b2289399949b584b34089f7e4f75e2f1763\": container with ID starting with 0fc25ddcb9028983f0fa28caede55b2289399949b584b34089f7e4f75e2f1763 not found: ID does not exist"
Oct 30 13:11:32.550531 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.549731    5893 scope.go:122] "RemoveContainer" containerID="7da42dca14ce2b47c076bb91da94c1b5622306cf0a0072dc5047969d268ea1a7"
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.573546    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-data\") pod \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.573665    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-workload-certs\") pod \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.573731    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-credential-socket\") pod \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.573761    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-credential-socket\") pod \"a39cee14-750a-47d1-8b5c-d362c150a6d1\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.573795    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-workload-socket\") pod \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.573863    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-workload-socket\") pod \"a39cee14-750a-47d1-8b5c-d362c150a6d1\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.573891    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a39cee14-750a-47d1-8b5c-d362c150a6d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39cee14-750a-47d1-8b5c-d362c150a6d1-istiod-ca-cert\") pod \"a39cee14-750a-47d1-8b5c-d362c150a6d1\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.573948    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-workload-certs\") pod \"a39cee14-750a-47d1-8b5c-d362c150a6d1\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.573987    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-podinfo\") pod \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.574023    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-token\" (UniqueName: \"kubernetes.io/projected/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-token\") pod \"a39cee14-750a-47d1-8b5c-d362c150a6d1\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.574078    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-kube-api-access-fkbs2\" (UniqueName: \"kubernetes.io/projected/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-kube-api-access-fkbs2\") pod \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.574117    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-envoy\") pod \"a39cee14-750a-47d1-8b5c-d362c150a6d1\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.574150    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-token\" (UniqueName: \"kubernetes.io/projected/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-token\") pod \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.574229    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a39cee14-750a-47d1-8b5c-d362c150a6d1-kube-api-access-wsbp9\" (UniqueName: \"kubernetes.io/projected/a39cee14-750a-47d1-8b5c-d362c150a6d1-kube-api-access-wsbp9\") pod \"a39cee14-750a-47d1-8b5c-d362c150a6d1\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.574331    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-data\") pod \"a39cee14-750a-47d1-8b5c-d362c150a6d1\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.574413    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-envoy\") pod \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.574474    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istiod-ca-cert\") pod \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\" (UID: \"42abda8f-3710-4edc-8bbb-f032aa9fa4f6\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.574532    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-podinfo\") pod \"a39cee14-750a-47d1-8b5c-d362c150a6d1\" (UID: \"a39cee14-750a-47d1-8b5c-d362c150a6d1\") "
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.574651    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a39cee14-750a-47d1-8b5c-d362c150a6d1-istiod-ca-cert" pod "a39cee14-750a-47d1-8b5c-d362c150a6d1" (UID: "a39cee14-750a-47d1-8b5c-d362c150a6d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.574918    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-data" pod "42abda8f-3710-4edc-8bbb-f032aa9fa4f6" (UID: "42abda8f-3710-4edc-8bbb-f032aa9fa4f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575263    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575306    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575322    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58cq6\" (UniqueName: \"kubernetes.io/projected/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-kube-api-access-58cq6\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575338    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575352    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575365    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575409    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a39cee14-750a-47d1-8b5c-d362c150a6d1-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575423    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.575413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575436    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575450    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575463    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e200c2e3-c88e-4707-a139-c5b8dfab3941-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575475    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575487    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575498    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575511    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575524    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575539    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwdd4\" (UniqueName: \"kubernetes.io/projected/e200c2e3-c88e-4707-a139-c5b8dfab3941-kube-api-access-bwdd4\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575552    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575565    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e200c2e3-c88e-4707-a139-c5b8dfab3941-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575578    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575768    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-workload-certs" pod "42abda8f-3710-4edc-8bbb-f032aa9fa4f6" (UID: "42abda8f-3710-4edc-8bbb-f032aa9fa4f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.575955    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-credential-socket" pod "42abda8f-3710-4edc-8bbb-f032aa9fa4f6" (UID: "42abda8f-3710-4edc-8bbb-f032aa9fa4f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.576089    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-credential-socket" pod "a39cee14-750a-47d1-8b5c-d362c150a6d1" (UID: "a39cee14-750a-47d1-8b5c-d362c150a6d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.576209    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-workload-socket" pod "42abda8f-3710-4edc-8bbb-f032aa9fa4f6" (UID: "42abda8f-3710-4edc-8bbb-f032aa9fa4f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.576975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.576357    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-workload-socket" pod "a39cee14-750a-47d1-8b5c-d362c150a6d1" (UID: "a39cee14-750a-47d1-8b5c-d362c150a6d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.578326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.577991    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-workload-certs" pod "a39cee14-750a-47d1-8b5c-d362c150a6d1" (UID: "a39cee14-750a-47d1-8b5c-d362c150a6d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.580677 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.580380    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-podinfo" pod "42abda8f-3710-4edc-8bbb-f032aa9fa4f6" (UID: "42abda8f-3710-4edc-8bbb-f032aa9fa4f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:11:32.582853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.582586    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-token" pod "a39cee14-750a-47d1-8b5c-d362c150a6d1" (UID: "a39cee14-750a-47d1-8b5c-d362c150a6d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:32.584694 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.584671    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-kube-api-access-fkbs2" pod "42abda8f-3710-4edc-8bbb-f032aa9fa4f6" (UID: "42abda8f-3710-4edc-8bbb-f032aa9fa4f6"). InnerVolumeSpecName "kube-api-access-fkbs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:32.586561 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.586539    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-envoy" pod "a39cee14-750a-47d1-8b5c-d362c150a6d1" (UID: "a39cee14-750a-47d1-8b5c-d362c150a6d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.588312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.588286    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-token" pod "42abda8f-3710-4edc-8bbb-f032aa9fa4f6" (UID: "42abda8f-3710-4edc-8bbb-f032aa9fa4f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:32.590381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.589556    5893 scope.go:122] "RemoveContainer" containerID="8fae83a6c49bbbacfb3e2d63963639842ed9819299fe06fa2f163581b6b87bd4"
Oct 30 13:11:32.590470 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.590390    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istiod-ca-cert" pod "42abda8f-3710-4edc-8bbb-f032aa9fa4f6" (UID: "42abda8f-3710-4edc-8bbb-f032aa9fa4f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:32.590871 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.590632    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-data" pod "a39cee14-750a-47d1-8b5c-d362c150a6d1" (UID: "a39cee14-750a-47d1-8b5c-d362c150a6d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.594668 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.592736    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a39cee14-750a-47d1-8b5c-d362c150a6d1-kube-api-access-wsbp9" pod "a39cee14-750a-47d1-8b5c-d362c150a6d1" (UID: "a39cee14-750a-47d1-8b5c-d362c150a6d1"). InnerVolumeSpecName "kube-api-access-wsbp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:32.596244 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.596196    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-podinfo" pod "a39cee14-750a-47d1-8b5c-d362c150a6d1" (UID: "a39cee14-750a-47d1-8b5c-d362c150a6d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:11:32.596787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.596754    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-envoy" pod "42abda8f-3710-4edc-8bbb-f032aa9fa4f6" (UID: "42abda8f-3710-4edc-8bbb-f032aa9fa4f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.599797 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.599773    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"]
Oct 30 13:11:32.604859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.604501    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-17fc433e-openshift-default-79d8c9969-pq6q7"]
Oct 30 13:11:32.612819 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.612799    5893 scope.go:122] "RemoveContainer" containerID="8fae83a6c49bbbacfb3e2d63963639842ed9819299fe06fa2f163581b6b87bd4"
Oct 30 13:11:32.613368 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:11:32.613234    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fae83a6c49bbbacfb3e2d63963639842ed9819299fe06fa2f163581b6b87bd4\": container with ID starting with 8fae83a6c49bbbacfb3e2d63963639842ed9819299fe06fa2f163581b6b87bd4 not found: ID does not exist" containerID="8fae83a6c49bbbacfb3e2d63963639842ed9819299fe06fa2f163581b6b87bd4"
Oct 30 13:11:32.613368 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.613301    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fae83a6c49bbbacfb3e2d63963639842ed9819299fe06fa2f163581b6b87bd4"} err="failed to get container status \"8fae83a6c49bbbacfb3e2d63963639842ed9819299fe06fa2f163581b6b87bd4\": rpc error: code = NotFound desc = could not find container \"8fae83a6c49bbbacfb3e2d63963639842ed9819299fe06fa2f163581b6b87bd4\": container with ID starting with 8fae83a6c49bbbacfb3e2d63963639842ed9819299fe06fa2f163581b6b87bd4 not found: ID does not exist"
Oct 30 13:11:32.699089 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.699001    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsbp9\" (UniqueName: \"kubernetes.io/projected/a39cee14-750a-47d1-8b5c-d362c150a6d1-kube-api-access-wsbp9\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.701359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.701335    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.701554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.701540    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.701666 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.701654    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.701755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.701744    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.701843 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.701831    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.701926 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.701913    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.702011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.701999    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.702087 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.702076    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.702288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.702166    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.702288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.702183    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.702288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.702196    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.702288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.702208    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.702288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.702223    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkbs2\" (UniqueName: \"kubernetes.io/projected/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-kube-api-access-fkbs2\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.702288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.702236    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a39cee14-750a-47d1-8b5c-d362c150a6d1-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.702288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.702249    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42abda8f-3710-4edc-8bbb-f032aa9fa4f6-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.778298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.777644    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"]
Oct 30 13:11:32.778298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.777688    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-mczbb"]
Oct 30 13:11:32.813916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.813891    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:11:32.853436 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.853401    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"]
Oct 30 13:11:32.857786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.857760    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-cmnxp"]
Oct 30 13:11:32.907357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.907267    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-credential-socket\") pod \"534048b5-3bb5-4eab-9357-9a464373036b\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") "
Oct 30 13:11:32.907598 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.907375    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534048b5-3bb5-4eab-9357-9a464373036b-istio-token\" (UniqueName: \"kubernetes.io/projected/534048b5-3bb5-4eab-9357-9a464373036b-istio-token\") pod \"534048b5-3bb5-4eab-9357-9a464373036b\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") "
Oct 30 13:11:32.907598 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.907466    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-istio-envoy\") pod \"534048b5-3bb5-4eab-9357-9a464373036b\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") "
Oct 30 13:11:32.907598 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.907526    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-workload-socket\") pod \"534048b5-3bb5-4eab-9357-9a464373036b\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") "
Oct 30 13:11:32.907598 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.907587    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/534048b5-3bb5-4eab-9357-9a464373036b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534048b5-3bb5-4eab-9357-9a464373036b-istiod-ca-cert\") pod \"534048b5-3bb5-4eab-9357-9a464373036b\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") "
Oct 30 13:11:32.907810 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.907618    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/534048b5-3bb5-4eab-9357-9a464373036b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534048b5-3bb5-4eab-9357-9a464373036b-istio-podinfo\") pod \"534048b5-3bb5-4eab-9357-9a464373036b\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") "
Oct 30 13:11:32.907810 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.907668    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534048b5-3bb5-4eab-9357-9a464373036b-kube-api-access-9rtlv\" (UniqueName: \"kubernetes.io/projected/534048b5-3bb5-4eab-9357-9a464373036b-kube-api-access-9rtlv\") pod \"534048b5-3bb5-4eab-9357-9a464373036b\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") "
Oct 30 13:11:32.907903 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.907856    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-istio-data\") pod \"534048b5-3bb5-4eab-9357-9a464373036b\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") "
Oct 30 13:11:32.907968 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.907948    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-workload-certs\") pod \"534048b5-3bb5-4eab-9357-9a464373036b\" (UID: \"534048b5-3bb5-4eab-9357-9a464373036b\") "
Oct 30 13:11:32.908253 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.908226    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-credential-socket" pod "534048b5-3bb5-4eab-9357-9a464373036b" (UID: "534048b5-3bb5-4eab-9357-9a464373036b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.909443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.908585    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-istio-data" pod "534048b5-3bb5-4eab-9357-9a464373036b" (UID: "534048b5-3bb5-4eab-9357-9a464373036b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.909443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.908747    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-workload-socket" pod "534048b5-3bb5-4eab-9357-9a464373036b" (UID: "534048b5-3bb5-4eab-9357-9a464373036b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.909443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.908849    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-workload-certs" pod "534048b5-3bb5-4eab-9357-9a464373036b" (UID: "534048b5-3bb5-4eab-9357-9a464373036b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:32.909443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.909234    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.909443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.909257    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.909443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.909290    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.909443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.909304    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:32.909443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.909302    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/534048b5-3bb5-4eab-9357-9a464373036b-istiod-ca-cert" pod "534048b5-3bb5-4eab-9357-9a464373036b" (UID: "534048b5-3bb5-4eab-9357-9a464373036b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:11:32.913192 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.913159    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534048b5-3bb5-4eab-9357-9a464373036b-istio-token" pod "534048b5-3bb5-4eab-9357-9a464373036b" (UID: "534048b5-3bb5-4eab-9357-9a464373036b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:32.913342 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.913204    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534048b5-3bb5-4eab-9357-9a464373036b-kube-api-access-9rtlv" pod "534048b5-3bb5-4eab-9357-9a464373036b" (UID: "534048b5-3bb5-4eab-9357-9a464373036b"). InnerVolumeSpecName "kube-api-access-9rtlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:32.913601 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.913575    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/534048b5-3bb5-4eab-9357-9a464373036b-istio-podinfo" pod "534048b5-3bb5-4eab-9357-9a464373036b" (UID: "534048b5-3bb5-4eab-9357-9a464373036b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:11:32.913698 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:32.913674    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-istio-envoy" pod "534048b5-3bb5-4eab-9357-9a464373036b" (UID: "534048b5-3bb5-4eab-9357-9a464373036b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:11:33.011243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.011203    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/534048b5-3bb5-4eab-9357-9a464373036b-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:33.011243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.011242    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/534048b5-3bb5-4eab-9357-9a464373036b-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:33.011665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.011257    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rtlv\" (UniqueName: \"kubernetes.io/projected/534048b5-3bb5-4eab-9357-9a464373036b-kube-api-access-9rtlv\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:33.011665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.011292    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/534048b5-3bb5-4eab-9357-9a464373036b-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:33.011665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.011306    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/534048b5-3bb5-4eab-9357-9a464373036b-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:11:33.518259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.518233    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"
Oct 30 13:11:33.518259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.518233    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n" event={"ID":"a39cee14-750a-47d1-8b5c-d362c150a6d1","Type":"ContainerDied","Data":"5ed3d21db12db8797ff92299ab30afa0759c2b10830d2bccc3ebe6641ea299ac"}
Oct 30 13:11:33.518570 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.518350    5893 scope.go:122] "RemoveContainer" containerID="e56c7ffefc4fb93b75b2112f3a71f0604204fb7097d50c467780e1e79b0bfdf3"
Oct 30 13:11:33.520897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.520863    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64" event={"ID":"534048b5-3bb5-4eab-9357-9a464373036b","Type":"ContainerDied","Data":"fe256e901fd28e351f0bee2ff02399f1d28997ba1ab5f6038030dfe0dc9ffe1e"}
Oct 30 13:11:33.521253 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.521233    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"
Oct 30 13:11:33.533999 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.533975    5893 scope.go:122] "RemoveContainer" containerID="c5aecc44f4e5e95ec87e8fda8655e05a6dc5aa6ff034b5a786a93d0f283057d6"
Oct 30 13:11:33.592340 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.592307    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"]
Oct 30 13:11:33.607511 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.607482    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-d5t8n"]
Oct 30 13:11:33.666117 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.666082    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"]
Oct 30 13:11:33.683129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:33.683100    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-67f56e-openshift-default-564fb6b9d7-c6l64"]
Oct 30 13:11:34.203461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.203421    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"]
Oct 30 13:11:34.204361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204343    5893 cpu_manager.go:414] "containerMap: removing container" podUID="a39cee14-750a-47d1-8b5c-d362c150a6d1" containerName="istio-proxy"
Oct 30 13:11:34.204361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204360    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a39cee14-750a-47d1-8b5c-d362c150a6d1" containerName="istio-proxy"
Oct 30 13:11:34.204487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204382    5893 cpu_manager.go:414] "containerMap: removing container" podUID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerName="istio-proxy"
Oct 30 13:11:34.204487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204388    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerName="istio-proxy"
Oct 30 13:11:34.204487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204395    5893 cpu_manager.go:414] "containerMap: removing container" podUID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerName="istio-proxy"
Oct 30 13:11:34.204487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204399    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerName="istio-proxy"
Oct 30 13:11:34.204487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204412    5893 cpu_manager.go:414] "containerMap: removing container" podUID="534048b5-3bb5-4eab-9357-9a464373036b" containerName="istio-proxy"
Oct 30 13:11:34.204487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204416    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534048b5-3bb5-4eab-9357-9a464373036b" containerName="istio-proxy"
Oct 30 13:11:34.204487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204432    5893 cpu_manager.go:414] "containerMap: removing container" podUID="e200c2e3-c88e-4707-a139-c5b8dfab3941" containerName="istio-proxy"
Oct 30 13:11:34.204487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204437    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e200c2e3-c88e-4707-a139-c5b8dfab3941" containerName="istio-proxy"
Oct 30 13:11:34.204487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204454    5893 cpu_manager.go:414] "containerMap: removing container" podUID="42abda8f-3710-4edc-8bbb-f032aa9fa4f6" containerName="istio-proxy"
Oct 30 13:11:34.204487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204458    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42abda8f-3710-4edc-8bbb-f032aa9fa4f6" containerName="istio-proxy"
Oct 30 13:11:34.204809 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204559    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="a39cee14-750a-47d1-8b5c-d362c150a6d1" containerName="istio-proxy"
Oct 30 13:11:34.204809 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204566    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerName="istio-proxy"
Oct 30 13:11:34.204809 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204572    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" containerName="istio-proxy"
Oct 30 13:11:34.204809 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204581    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="534048b5-3bb5-4eab-9357-9a464373036b" containerName="istio-proxy"
Oct 30 13:11:34.204809 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204590    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="e200c2e3-c88e-4707-a139-c5b8dfab3941" containerName="istio-proxy"
Oct 30 13:11:34.204809 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.204597    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="42abda8f-3710-4edc-8bbb-f032aa9fa4f6" containerName="istio-proxy"
Oct 30 13:11:34.210091 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.210063    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"
Oct 30 13:11:34.229957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.229842    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1c0e6c42-dockercfg-9mgj7\""
Oct 30 13:11:34.229957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.229928    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1c0e6c42\""
Oct 30 13:11:34.232693 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.232665    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"]
Oct 30 13:11:34.343296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.337746    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08ff971-0181-4107-ad1c-61f9b0ebc70b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9\" (UID: \"a08ff971-0181-4107-ad1c-61f9b0ebc70b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"
Oct 30 13:11:34.343296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.337805    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv64z\" (UniqueName: \"kubernetes.io/projected/a08ff971-0181-4107-ad1c-61f9b0ebc70b-kube-api-access-bv64z\") pod \"payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9\" (UID: \"a08ff971-0181-4107-ad1c-61f9b0ebc70b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"
Oct 30 13:11:34.354175 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.351781    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"]
Oct 30 13:11:34.378299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.360249    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"
Oct 30 13:11:34.407313 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.397004    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"]
Oct 30 13:11:34.454347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.448649    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08ff971-0181-4107-ad1c-61f9b0ebc70b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9\" (UID: \"a08ff971-0181-4107-ad1c-61f9b0ebc70b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"
Oct 30 13:11:34.454347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.448703    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv64z\" (UniqueName: \"kubernetes.io/projected/a08ff971-0181-4107-ad1c-61f9b0ebc70b-kube-api-access-bv64z\") pod \"payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9\" (UID: \"a08ff971-0181-4107-ad1c-61f9b0ebc70b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"
Oct 30 13:11:34.454347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.450017    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08ff971-0181-4107-ad1c-61f9b0ebc70b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9\" (UID: \"a08ff971-0181-4107-ad1c-61f9b0ebc70b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"
Oct 30 13:11:34.508353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.504200    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv64z\" (UniqueName: \"kubernetes.io/projected/a08ff971-0181-4107-ad1c-61f9b0ebc70b-kube-api-access-bv64z\") pod \"payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9\" (UID: \"a08ff971-0181-4107-ad1c-61f9b0ebc70b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"
Oct 30 13:11:34.591860 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.591546    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"
Oct 30 13:11:34.601010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.598586    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xhn\" (UniqueName: \"kubernetes.io/projected/d05d155d-5f05-4945-9d25-4be97a2420e1-kube-api-access-s5xhn\") pod \"payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5\" (UID: \"d05d155d-5f05-4945-9d25-4be97a2420e1\") " pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"
Oct 30 13:11:34.601010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.598745    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05d155d-5f05-4945-9d25-4be97a2420e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5\" (UID: \"d05d155d-5f05-4945-9d25-4be97a2420e1\") " pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"
Oct 30 13:11:34.670957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.670921    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42abda8f-3710-4edc-8bbb-f032aa9fa4f6" path="/var/lib/kubelet/pods/42abda8f-3710-4edc-8bbb-f032aa9fa4f6/volumes"
Oct 30 13:11:34.671668 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.671646    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="534048b5-3bb5-4eab-9357-9a464373036b" path="/var/lib/kubelet/pods/534048b5-3bb5-4eab-9357-9a464373036b/volumes"
Oct 30 13:11:34.672347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.672321    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a39cee14-750a-47d1-8b5c-d362c150a6d1" path="/var/lib/kubelet/pods/a39cee14-750a-47d1-8b5c-d362c150a6d1/volumes"
Oct 30 13:11:34.673399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.673376    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e200c2e3-c88e-4707-a139-c5b8dfab3941" path="/var/lib/kubelet/pods/e200c2e3-c88e-4707-a139-c5b8dfab3941/volumes"
Oct 30 13:11:34.674012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.673993    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7" path="/var/lib/kubelet/pods/f0b1e4e2-ea57-4b68-b2bb-0a0de945c8a7/volumes"
Oct 30 13:11:34.701037 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.700989    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xhn\" (UniqueName: \"kubernetes.io/projected/d05d155d-5f05-4945-9d25-4be97a2420e1-kube-api-access-s5xhn\") pod \"payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5\" (UID: \"d05d155d-5f05-4945-9d25-4be97a2420e1\") " pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"
Oct 30 13:11:34.701200 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.701179    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05d155d-5f05-4945-9d25-4be97a2420e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5\" (UID: \"d05d155d-5f05-4945-9d25-4be97a2420e1\") " pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"
Oct 30 13:11:34.702510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.702487    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05d155d-5f05-4945-9d25-4be97a2420e1-plugins-config-volume\") pod \"payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5\" (UID: \"d05d155d-5f05-4945-9d25-4be97a2420e1\") " pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"
Oct 30 13:11:34.729085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.729022    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xhn\" (UniqueName: \"kubernetes.io/projected/d05d155d-5f05-4945-9d25-4be97a2420e1-kube-api-access-s5xhn\") pod \"payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5\" (UID: \"d05d155d-5f05-4945-9d25-4be97a2420e1\") " pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"
Oct 30 13:11:34.749685 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:34.749325    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"
Oct 30 13:11:35.003730 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:35.003701    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"]
Oct 30 13:11:35.004285 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:11:35.004250    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08ff971_0181_4107_ad1c_61f9b0ebc70b.slice/crio-afd9f409d9182df724c064fea3024f066e8b52375a75b38c6c3ea1a8d3e9af8a WatchSource:0}: Error finding container afd9f409d9182df724c064fea3024f066e8b52375a75b38c6c3ea1a8d3e9af8a: Status 404 returned error can't find the container with id afd9f409d9182df724c064fea3024f066e8b52375a75b38c6c3ea1a8d3e9af8a
Oct 30 13:11:35.114909 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:35.114882    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"]
Oct 30 13:11:35.117050 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:11:35.117023    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05d155d_5f05_4945_9d25_4be97a2420e1.slice/crio-2f313ea03e9faba44a6dedb14acec22567bc43e34a1a6ca86df7c44206bb4321 WatchSource:0}: Error finding container 2f313ea03e9faba44a6dedb14acec22567bc43e34a1a6ca86df7c44206bb4321: Status 404 returned error can't find the container with id 2f313ea03e9faba44a6dedb14acec22567bc43e34a1a6ca86df7c44206bb4321
Oct 30 13:11:35.679100 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:35.679060    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5" event={"ID":"d05d155d-5f05-4945-9d25-4be97a2420e1","Type":"ContainerStarted","Data":"e94432ce22a9f10cafe9b51c0ca1f23d9771270ed352ab4e04de775863a08a23"}
Oct 30 13:11:35.679627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:35.679130    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5" event={"ID":"d05d155d-5f05-4945-9d25-4be97a2420e1","Type":"ContainerStarted","Data":"2f313ea03e9faba44a6dedb14acec22567bc43e34a1a6ca86df7c44206bb4321"}
Oct 30 13:11:35.679627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:35.679306    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"
Oct 30 13:11:35.681143 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:35.681113    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9" event={"ID":"a08ff971-0181-4107-ad1c-61f9b0ebc70b","Type":"ContainerStarted","Data":"a4824a8359d4b13ce3bfe038056bf68646c6d1f5a31bc0e1d032d5df0e0d6cf9"}
Oct 30 13:11:35.681317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:35.681156    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9" event={"ID":"a08ff971-0181-4107-ad1c-61f9b0ebc70b","Type":"ContainerStarted","Data":"afd9f409d9182df724c064fea3024f066e8b52375a75b38c6c3ea1a8d3e9af8a"}
Oct 30 13:11:35.681317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:35.681191    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"
Oct 30 13:11:35.724444 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:35.724396    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5" podStartSLOduration=1.724381494 podStartE2EDuration="1.724381494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:35.721078294 +0000 UTC m=+1537.304507985" watchObservedRunningTime="2026-10-30 13:11:35.724381494 +0000 UTC m=+1537.307811184"
Oct 30 13:11:35.767712 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:35.767653    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9" podStartSLOduration=1.7676343920000002 podStartE2EDuration="1.767634392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:35.762682931 +0000 UTC m=+1537.346112637" watchObservedRunningTime="2026-10-30 13:11:35.767634392 +0000 UTC m=+1537.351064086"
Oct 30 13:11:36.308148 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.308120    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"]
Oct 30 13:11:36.312695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.312678    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"
Oct 30 13:11:36.338237 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.338206    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"]
Oct 30 13:11:36.338422 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.338396    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-abbc6c\""
Oct 30 13:11:36.411298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.406533    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-abbc6c-dockercfg-9ddm4\""
Oct 30 13:11:36.426584 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.420407    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"]
Oct 30 13:11:36.461980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.459397    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d261ff50-28dc-4622-8dae-75501cf12028-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz\" (UID: \"d261ff50-28dc-4622-8dae-75501cf12028\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"
Oct 30 13:11:36.461980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.459511    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czsgf\" (UniqueName: \"kubernetes.io/projected/d261ff50-28dc-4622-8dae-75501cf12028-kube-api-access-czsgf\") pod \"payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz\" (UID: \"d261ff50-28dc-4622-8dae-75501cf12028\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"
Oct 30 13:11:36.490636 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.488530    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"
Oct 30 13:11:36.512163 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.511384    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"]
Oct 30 13:11:36.597981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.596746    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czsgf\" (UniqueName: \"kubernetes.io/projected/d261ff50-28dc-4622-8dae-75501cf12028-kube-api-access-czsgf\") pod \"payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz\" (UID: \"d261ff50-28dc-4622-8dae-75501cf12028\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"
Oct 30 13:11:36.597981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.596854    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0769c09e-fccd-4411-9a8f-824acd1dcd27-plugins-config-volume\") pod \"payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl\" (UID: \"0769c09e-fccd-4411-9a8f-824acd1dcd27\") " pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"
Oct 30 13:11:36.597981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.596938    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxg6\" (UniqueName: \"kubernetes.io/projected/0769c09e-fccd-4411-9a8f-824acd1dcd27-kube-api-access-6hxg6\") pod \"payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl\" (UID: \"0769c09e-fccd-4411-9a8f-824acd1dcd27\") " pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"
Oct 30 13:11:36.597981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.597019    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d261ff50-28dc-4622-8dae-75501cf12028-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz\" (UID: \"d261ff50-28dc-4622-8dae-75501cf12028\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"
Oct 30 13:11:36.597981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.597880    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d261ff50-28dc-4622-8dae-75501cf12028-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz\" (UID: \"d261ff50-28dc-4622-8dae-75501cf12028\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"
Oct 30 13:11:36.658390 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.654945    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czsgf\" (UniqueName: \"kubernetes.io/projected/d261ff50-28dc-4622-8dae-75501cf12028-kube-api-access-czsgf\") pod \"payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz\" (UID: \"d261ff50-28dc-4622-8dae-75501cf12028\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"
Oct 30 13:11:36.698348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.698305    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0769c09e-fccd-4411-9a8f-824acd1dcd27-plugins-config-volume\") pod \"payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl\" (UID: \"0769c09e-fccd-4411-9a8f-824acd1dcd27\") " pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"
Oct 30 13:11:36.699253 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.699224    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxg6\" (UniqueName: \"kubernetes.io/projected/0769c09e-fccd-4411-9a8f-824acd1dcd27-kube-api-access-6hxg6\") pod \"payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl\" (UID: \"0769c09e-fccd-4411-9a8f-824acd1dcd27\") " pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"
Oct 30 13:11:36.699924 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.699869    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0769c09e-fccd-4411-9a8f-824acd1dcd27-plugins-config-volume\") pod \"payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl\" (UID: \"0769c09e-fccd-4411-9a8f-824acd1dcd27\") " pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"
Oct 30 13:11:36.722249 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.722216    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxg6\" (UniqueName: \"kubernetes.io/projected/0769c09e-fccd-4411-9a8f-824acd1dcd27-kube-api-access-6hxg6\") pod \"payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl\" (UID: \"0769c09e-fccd-4411-9a8f-824acd1dcd27\") " pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"
Oct 30 13:11:36.854498 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.854417    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"]
Oct 30 13:11:36.860148 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.860124    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:36.876011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.875956    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-e9d641-serving-cert\""
Oct 30 13:11:36.881218 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.881191    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"]
Oct 30 13:11:36.914837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.914811    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"
Oct 30 13:11:36.929624 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:36.929468    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"
Oct 30 13:11:37.004887 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.004135    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00423102-32b5-4a2a-8f6d-e93309aae3b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:37.004887 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.004187    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zbhr\" (UniqueName: \"kubernetes.io/projected/00423102-32b5-4a2a-8f6d-e93309aae3b6-kube-api-access-4zbhr\") pod \"maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:37.004887 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.004370    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00423102-32b5-4a2a-8f6d-e93309aae3b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:37.004887 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.004402    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00423102-32b5-4a2a-8f6d-e93309aae3b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:37.108587 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.108482    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00423102-32b5-4a2a-8f6d-e93309aae3b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:37.108587 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.108556    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00423102-32b5-4a2a-8f6d-e93309aae3b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:37.108815 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.108800    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00423102-32b5-4a2a-8f6d-e93309aae3b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:37.108876 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.108860    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zbhr\" (UniqueName: \"kubernetes.io/projected/00423102-32b5-4a2a-8f6d-e93309aae3b6-kube-api-access-4zbhr\") pod \"maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:37.119538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.119225    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00423102-32b5-4a2a-8f6d-e93309aae3b6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:37.132552 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.122369    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00423102-32b5-4a2a-8f6d-e93309aae3b6-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:37.137206 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.136702    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zbhr\" (UniqueName: \"kubernetes.io/projected/00423102-32b5-4a2a-8f6d-e93309aae3b6-kube-api-access-4zbhr\") pod \"maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:37.137206 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.137164    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00423102-32b5-4a2a-8f6d-e93309aae3b6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:37.190932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.189689    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:37.267003 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.265168    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"]
Oct 30 13:11:37.270050 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:11:37.270017    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0769c09e_fccd_4411_9a8f_824acd1dcd27.slice/crio-37fc5e331ab019e64c6142d8a2b9890512522169daca6492a198bc2917f462f4 WatchSource:0}: Error finding container 37fc5e331ab019e64c6142d8a2b9890512522169daca6492a198bc2917f462f4: Status 404 returned error can't find the container with id 37fc5e331ab019e64c6142d8a2b9890512522169daca6492a198bc2917f462f4
Oct 30 13:11:37.297296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.297247    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"]
Oct 30 13:11:37.523853 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:11:37.523814    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00423102_32b5_4a2a_8f6d_e93309aae3b6.slice/crio-e4dfe804d014738d619e155e440fd1825af55b8e1dd54d1a3393f0a93625dd13 WatchSource:0}: Error finding container e4dfe804d014738d619e155e440fd1825af55b8e1dd54d1a3393f0a93625dd13: Status 404 returned error can't find the container with id e4dfe804d014738d619e155e440fd1825af55b8e1dd54d1a3393f0a93625dd13
Oct 30 13:11:37.524596 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.524573    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"]
Oct 30 13:11:37.525669 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.525652    5893 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:11:37.717411 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.717262    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz" event={"ID":"d261ff50-28dc-4622-8dae-75501cf12028","Type":"ContainerStarted","Data":"b899adcbf99a135deb171b52fee02722eff5c2b8d3750e3cfa9d87a68d321104"}
Oct 30 13:11:37.717411 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.717424    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"
Oct 30 13:11:37.717411 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.717452    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz" event={"ID":"d261ff50-28dc-4622-8dae-75501cf12028","Type":"ContainerStarted","Data":"ff954e22285f7aea89ea6fcfdc700d2860948a46c503d3d42e846ffbefc9b340"}
Oct 30 13:11:37.719419 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.719388    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5" event={"ID":"00423102-32b5-4a2a-8f6d-e93309aae3b6","Type":"ContainerStarted","Data":"e4dfe804d014738d619e155e440fd1825af55b8e1dd54d1a3393f0a93625dd13"}
Oct 30 13:11:37.725032 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.721716    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl" event={"ID":"0769c09e-fccd-4411-9a8f-824acd1dcd27","Type":"ContainerStarted","Data":"cd1335ba203affc8174f23dee6d48359c599728355ab7e5d243365955a86ff8c"}
Oct 30 13:11:37.725032 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.721778    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl" event={"ID":"0769c09e-fccd-4411-9a8f-824acd1dcd27","Type":"ContainerStarted","Data":"37fc5e331ab019e64c6142d8a2b9890512522169daca6492a198bc2917f462f4"}
Oct 30 13:11:37.725032 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.722094    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"
Oct 30 13:11:37.764295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.762176    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz" podStartSLOduration=1.7621558 podStartE2EDuration="1.7621558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:37.757694679 +0000 UTC m=+1539.341124392" watchObservedRunningTime="2026-10-30 13:11:37.7621558 +0000 UTC m=+1539.345585495"
Oct 30 13:11:37.803026 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:37.802966    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl" podStartSLOduration=1.802947961 podStartE2EDuration="1.802947961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:37.799116982 +0000 UTC m=+1539.382546704" watchObservedRunningTime="2026-10-30 13:11:37.802947961 +0000 UTC m=+1539.386377656"
Oct 30 13:11:38.730421 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.730336    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5" event={"ID":"00423102-32b5-4a2a-8f6d-e93309aae3b6","Type":"ContainerStarted","Data":"545a93a7e800b4f316819642d23eab81192e2a5fe70ba70e127e3d728938a8ff"}
Oct 30 13:11:38.730421 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.730414    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:38.782489 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.782436    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5" podStartSLOduration=2.378023359 podStartE2EDuration="2.782415071s" totalImagesPullingTime="404.391712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:38.780087042 +0000 UTC m=+1540.363516735" watchObservedRunningTime="2026-10-30 13:11:38.782415071 +0000 UTC m=+1540.365844758"
Oct 30 13:11:38.808449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.808418    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"]
Oct 30 13:11:38.814386 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.814360    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:38.845783 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.845751    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"]
Oct 30 13:11:38.892627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.892597    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-716b4c-openshift-default-dockercfg-bhwx9\""
Oct 30 13:11:38.969294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.968751    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-envoy\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:38.969294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.968809    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-workload-socket\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:38.969294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.968912    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-workload-certs\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:38.969294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.969046    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-podinfo\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:38.969294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.969152    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-token\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:38.969294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.969213    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhnzf\" (UniqueName: \"kubernetes.io/projected/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-kube-api-access-lhnzf\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:38.969746 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.969318    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-credential-socket\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:38.969746 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.969364    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-data\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:38.969746 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:38.969403    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istiod-ca-cert\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.075347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.072451    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-workload-certs\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.075347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.072579    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-podinfo\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.075347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.072672    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-token\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.075347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.072726    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhnzf\" (UniqueName: \"kubernetes.io/projected/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-kube-api-access-lhnzf\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.075347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.072796    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-credential-socket\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.075347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.072840    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-data\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.075347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.072874    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istiod-ca-cert\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.075347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.072930    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-envoy\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.075347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.072961    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-workload-socket\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.075347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.073506    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-workload-socket\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.075347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.073844    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-workload-certs\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.075347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.074571    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-credential-socket\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.076054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.075692    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istiod-ca-cert\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.076054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.075939    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-data\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.122300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.112843    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-envoy\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.125051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.124985    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-podinfo\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.139936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.138524    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhnzf\" (UniqueName: \"kubernetes.io/projected/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-kube-api-access-lhnzf\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.141979 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.141825    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-token\") pod \"e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") " pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.431706 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.431623    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:39.782702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.782655    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"]
Oct 30 13:11:39.784714 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:11:39.784683    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec70a1d_58a8_4db9_ac0e_a7a08cefd63a.slice/crio-1a7bbc97be2967abe4335a16a4dad262c6c844323205f54bfdeb051a5826daa1 WatchSource:0}: Error finding container 1a7bbc97be2967abe4335a16a4dad262c6c844323205f54bfdeb051a5826daa1: Status 404 returned error can't find the container with id 1a7bbc97be2967abe4335a16a4dad262c6c844323205f54bfdeb051a5826daa1
Oct 30 13:11:39.787903 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.787867    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:39.788012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.787947    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:39.788012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:39.787988    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:40.748844 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:40.748803    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" event={"ID":"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a","Type":"ContainerStarted","Data":"23e2df560a06d0096ca32c0be19a7dd026b7b03e11c01a4967e34d44c277ac02"}
Oct 30 13:11:40.749036 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:40.748871    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" event={"ID":"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a","Type":"ContainerStarted","Data":"1a7bbc97be2967abe4335a16a4dad262c6c844323205f54bfdeb051a5826daa1"}
Oct 30 13:11:40.808330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:40.808257    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" podStartSLOduration=2.808241809 podStartE2EDuration="2.808241809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:40.804610133 +0000 UTC m=+1542.388039826" watchObservedRunningTime="2026-10-30 13:11:40.808241809 +0000 UTC m=+1542.391671502"
Oct 30 13:11:41.432025 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:41.431996    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:41.437816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:41.437788    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:41.758956 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:41.758925    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:41.759928 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:41.759907    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:11:44.743897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:44.743867    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:11:45.324761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:45.324553    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"]
Oct 30 13:11:45.455301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:45.446212    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"]
Oct 30 13:11:45.497663 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:45.489462    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"
Oct 30 13:11:45.503438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:45.503391    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013-plugins-config-volume\") pod \"payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q\" (UID: \"a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013\") " pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"
Oct 30 13:11:45.503815 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:45.503792    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k68dr\" (UniqueName: \"kubernetes.io/projected/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013-kube-api-access-k68dr\") pod \"payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q\" (UID: \"a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013\") " pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"
Oct 30 13:11:45.523076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:45.512653    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-716b4c-dockercfg-zw46q\""
Oct 30 13:11:45.523076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:45.513395    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-716b4c\""
Oct 30 13:11:45.606930 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:45.606839    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k68dr\" (UniqueName: \"kubernetes.io/projected/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013-kube-api-access-k68dr\") pod \"payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q\" (UID: \"a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013\") " pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"
Oct 30 13:11:45.607293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:45.607254    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013-plugins-config-volume\") pod \"payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q\" (UID: \"a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013\") " pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"
Oct 30 13:11:45.608079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:45.608056    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013-plugins-config-volume\") pod \"payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q\" (UID: \"a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013\") " pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"
Oct 30 13:11:45.631335 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:45.631300    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k68dr\" (UniqueName: \"kubernetes.io/projected/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013-kube-api-access-k68dr\") pod \"payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q\" (UID: \"a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013\") " pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"
Oct 30 13:11:45.890115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:45.890027    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"
Oct 30 13:11:46.243161 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:46.243129    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"]
Oct 30 13:11:46.246439 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:11:46.246404    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9cd3ca0_9231_47b7_8c2d_fb0b6ab88013.slice/crio-ab0608b2bab8a998ba7d25a23f95076c08bae0dad9f900020b524c895fe09c03 WatchSource:0}: Error finding container ab0608b2bab8a998ba7d25a23f95076c08bae0dad9f900020b524c895fe09c03: Status 404 returned error can't find the container with id ab0608b2bab8a998ba7d25a23f95076c08bae0dad9f900020b524c895fe09c03
Oct 30 13:11:46.696252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:46.696161    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"
Oct 30 13:11:46.696461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:46.696255    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"
Oct 30 13:11:46.837712 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:46.837680    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q" event={"ID":"a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013","Type":"ContainerStarted","Data":"11607827ab87fca40c1caaf805435fb971b0194f63590f554ae731c09f4ae64f"}
Oct 30 13:11:46.837906 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:46.837746    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q" event={"ID":"a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013","Type":"ContainerStarted","Data":"ab0608b2bab8a998ba7d25a23f95076c08bae0dad9f900020b524c895fe09c03"}
Oct 30 13:11:46.837906 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:46.837776    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"
Oct 30 13:11:46.879697 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:46.879639    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q" podStartSLOduration=1.8796215859999998 podStartE2EDuration="1.879621586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:46.878706676 +0000 UTC m=+1548.462136381" watchObservedRunningTime="2026-10-30 13:11:46.879621586 +0000 UTC m=+1548.463051281"
Oct 30 13:11:48.655139 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:11:48.655102    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:11:48.741180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:48.741153    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"
Oct 30 13:11:48.744373 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:48.742803    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"
Oct 30 13:11:54.401639 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.401599    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"]
Oct 30 13:11:54.407104 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.407078    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.423395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.423372    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-31e10b-openshift-default-dockercfg-9j4b6\""
Oct 30 13:11:54.439761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.439731    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"]
Oct 30 13:11:54.468826 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.468789    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-envoy\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.469045 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.468848    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-token\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.469114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.469061    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-workload-socket\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.469114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.469091    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-podinfo\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.469311 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.469250    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-credential-socket\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.469442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.469422    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-workload-certs\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.469498 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.469475    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxvw\" (UniqueName: \"kubernetes.io/projected/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-kube-api-access-xpxvw\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.469546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.469508    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-data\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.469595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.469563    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istiod-ca-cert\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.571045 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.571013    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-workload-certs\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.571239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.571077    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxvw\" (UniqueName: \"kubernetes.io/projected/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-kube-api-access-xpxvw\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.571239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.571113    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-data\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.571239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.571174    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istiod-ca-cert\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.571392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.571256    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-envoy\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.571392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.571307    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-token\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.571473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.571450    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-workload-socket\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.571527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.571478    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-podinfo\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.571575 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.571565    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-credential-socket\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.572142 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.572112    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-credential-socket\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.577075 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.574592    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-workload-socket\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.577075 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.574863    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-data\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.577075 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.575705    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istiod-ca-cert\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.577075 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.576991    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-workload-certs\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.580368 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.580042    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-envoy\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.597493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.597449    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-podinfo\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.599099 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.599042    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-token\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.599260 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.599146    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxvw\" (UniqueName: \"kubernetes.io/projected/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-kube-api-access-xpxvw\") pod \"e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") " pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:54.721466 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:54.721434    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:55.060117 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:55.060091    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"]
Oct 30 13:11:55.061009 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:11:55.060970    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6b0cb8_104f_47f6_abda_9f34c5c1c12e.slice/crio-ab7c0f9a34620044b93e17ce04d01324f9636b589f9d621aa2d5c19f98f53db7 WatchSource:0}: Error finding container ab7c0f9a34620044b93e17ce04d01324f9636b589f9d621aa2d5c19f98f53db7: Status 404 returned error can't find the container with id ab7c0f9a34620044b93e17ce04d01324f9636b589f9d621aa2d5c19f98f53db7
Oct 30 13:11:55.064175 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:55.064140    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:55.064266 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:55.064213    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:55.064266 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:55.064246    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:11:55.909853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:55.909819    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn" event={"ID":"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e","Type":"ContainerStarted","Data":"cb6e8e7285097ca05a81f20d35e3b89cbb2899d09edad40d05f6067b821a6ca5"}
Oct 30 13:11:55.910246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:55.909875    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn" event={"ID":"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e","Type":"ContainerStarted","Data":"ab7c0f9a34620044b93e17ce04d01324f9636b589f9d621aa2d5c19f98f53db7"}
Oct 30 13:11:56.722571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:56.722539    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:11:56.723989 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:56.723963    5893 patch_prober.go:29] interesting pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 13:11:56.724108 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:56.724014    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn" podUID="3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 13:11:56.733516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:56.733475    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn" podStartSLOduration=2.7334625839999998 podStartE2EDuration="2.733462584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:55.976696075 +0000 UTC m=+1557.560125770" watchObservedRunningTime="2026-10-30 13:11:56.733462584 +0000 UTC m=+1558.316892277"
Oct 30 13:11:56.737210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:56.737186    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"]
Oct 30 13:11:56.737418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:56.737399    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv" podUID="761ab5b8-a944-4b6a-acbc-327f8523ad5c" containerName="istio-proxy" containerID="cri-o://ab35bb80c9f018540e860f7b0ad59fba29fe679e9cf8f718e12a5729e6b4e222" gracePeriod=30
Oct 30 13:11:56.792312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:56.791352    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"]
Oct 30 13:11:56.792312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:56.791428    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"]
Oct 30 13:11:56.792312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:56.791613    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt" podUID="9bd32d4c-3366-434c-8917-862060b8eda4" containerName="istio-proxy" containerID="cri-o://85718abfee2812b0dd3dc4d45c870f81e4dbecbae2c6e210d5e87baae29d2f72" gracePeriod=30
Oct 30 13:11:56.792312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:56.791857    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp" podUID="40bb5401-70e8-4806-8739-ae835600784e" containerName="istio-proxy" containerID="cri-o://e7d936761dc30e6214efa57d8c9cc98ccd4341ab55966e195d51b743a59b516f" gracePeriod=30
Oct 30 13:11:57.724329 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:57.722159    5893 patch_prober.go:29] interesting pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 13:11:57.724329 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:57.722214    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn" podUID="3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 13:11:57.874516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:57.873085    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"
Oct 30 13:11:58.722115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:58.722078    5893 patch_prober.go:29] interesting pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 13:11:58.722314 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:58.722140    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn" podUID="3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 13:11:59.722332 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:59.722301    5893 patch_prober.go:29] interesting pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 13:11:59.722803 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:11:59.722359    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn" podUID="3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 13:12:00.722664 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:00.722624    5893 patch_prober.go:29] interesting pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Oct 30 13:12:00.723056 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:00.722685    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn" podUID="3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Oct 30 13:12:01.727330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:01.727296    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:12:01.727851 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:01.727835    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:12:01.728675 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:01.728651    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:12:01.985491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:01.985402    5893 generic.go:356] "Generic (PLEG): container finished" podID="9bd32d4c-3366-434c-8917-862060b8eda4" containerID="85718abfee2812b0dd3dc4d45c870f81e4dbecbae2c6e210d5e87baae29d2f72" exitCode=0
Oct 30 13:12:01.985650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:01.985608    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt" event={"ID":"9bd32d4c-3366-434c-8917-862060b8eda4","Type":"ContainerDied","Data":"85718abfee2812b0dd3dc4d45c870f81e4dbecbae2c6e210d5e87baae29d2f72"}
Oct 30 13:12:01.988050 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:01.987881    5893 generic.go:356] "Generic (PLEG): container finished" podID="761ab5b8-a944-4b6a-acbc-327f8523ad5c" containerID="ab35bb80c9f018540e860f7b0ad59fba29fe679e9cf8f718e12a5729e6b4e222" exitCode=0
Oct 30 13:12:01.988050 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:01.987964    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv" event={"ID":"761ab5b8-a944-4b6a-acbc-327f8523ad5c","Type":"ContainerDied","Data":"ab35bb80c9f018540e860f7b0ad59fba29fe679e9cf8f718e12a5729e6b4e222"}
Oct 30 13:12:01.988050 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:01.987999    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv" event={"ID":"761ab5b8-a944-4b6a-acbc-327f8523ad5c","Type":"ContainerDied","Data":"02ef4f541fae79598030a4266fb34c0ecfd84f3039f0182c7f5b303858cbe05e"}
Oct 30 13:12:01.988050 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:01.988015    5893 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02ef4f541fae79598030a4266fb34c0ecfd84f3039f0182c7f5b303858cbe05e"
Oct 30 13:12:01.990721 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:01.989845    5893 generic.go:356] "Generic (PLEG): container finished" podID="40bb5401-70e8-4806-8739-ae835600784e" containerID="e7d936761dc30e6214efa57d8c9cc98ccd4341ab55966e195d51b743a59b516f" exitCode=0
Oct 30 13:12:01.990721 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:01.990657    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp" event={"ID":"40bb5401-70e8-4806-8739-ae835600784e","Type":"ContainerDied","Data":"e7d936761dc30e6214efa57d8c9cc98ccd4341ab55966e195d51b743a59b516f"}
Oct 30 13:12:01.992203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:01.992183    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:12:02.049505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.049468    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istiod-ca-cert\") pod \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") "
Oct 30 13:12:02.049774 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.049755    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-workload-certs\") pod \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") "
Oct 30 13:12:02.049945 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.049930    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-token\" (UniqueName: \"kubernetes.io/projected/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-token\") pod \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") "
Oct 30 13:12:02.050072 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.050059    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-workload-socket\") pod \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") "
Oct 30 13:12:02.050235 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.050220    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-data\") pod \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") "
Oct 30 13:12:02.050434 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.050415    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/761ab5b8-a944-4b6a-acbc-327f8523ad5c-kube-api-access-r492t\" (UniqueName: \"kubernetes.io/projected/761ab5b8-a944-4b6a-acbc-327f8523ad5c-kube-api-access-r492t\") pod \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") "
Oct 30 13:12:02.050597 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.050583    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-credential-socket\") pod \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") "
Oct 30 13:12:02.050697 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.050684    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-podinfo\") pod \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") "
Oct 30 13:12:02.050816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.050803    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-envoy\") pod \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\" (UID: \"761ab5b8-a944-4b6a-acbc-327f8523ad5c\") "
Oct 30 13:12:02.054298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.050436    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-workload-certs" pod "761ab5b8-a944-4b6a-acbc-327f8523ad5c" (UID: "761ab5b8-a944-4b6a-acbc-327f8523ad5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.054298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.050742    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istiod-ca-cert" pod "761ab5b8-a944-4b6a-acbc-327f8523ad5c" (UID: "761ab5b8-a944-4b6a-acbc-327f8523ad5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:02.054298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.050955    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-data" pod "761ab5b8-a944-4b6a-acbc-327f8523ad5c" (UID: "761ab5b8-a944-4b6a-acbc-327f8523ad5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.054298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.054247    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-credential-socket" pod "761ab5b8-a944-4b6a-acbc-327f8523ad5c" (UID: "761ab5b8-a944-4b6a-acbc-327f8523ad5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.054817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.054798    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-workload-socket" pod "761ab5b8-a944-4b6a-acbc-327f8523ad5c" (UID: "761ab5b8-a944-4b6a-acbc-327f8523ad5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.058682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.058648    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-podinfo" pod "761ab5b8-a944-4b6a-acbc-327f8523ad5c" (UID: "761ab5b8-a944-4b6a-acbc-327f8523ad5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:12:02.058969 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.058934    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-envoy" pod "761ab5b8-a944-4b6a-acbc-327f8523ad5c" (UID: "761ab5b8-a944-4b6a-acbc-327f8523ad5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.059696 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.059671    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761ab5b8-a944-4b6a-acbc-327f8523ad5c-kube-api-access-r492t" pod "761ab5b8-a944-4b6a-acbc-327f8523ad5c" (UID: "761ab5b8-a944-4b6a-acbc-327f8523ad5c"). InnerVolumeSpecName "kube-api-access-r492t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:02.063575 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.063539    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-token" pod "761ab5b8-a944-4b6a-acbc-327f8523ad5c" (UID: "761ab5b8-a944-4b6a-acbc-327f8523ad5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:02.111581 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.111553    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:12:02.176034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.173730    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd32d4c-3366-434c-8917-862060b8eda4-kube-api-access-8r9cj\" (UniqueName: \"kubernetes.io/projected/9bd32d4c-3366-434c-8917-862060b8eda4-kube-api-access-8r9cj\") pod \"9bd32d4c-3366-434c-8917-862060b8eda4\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") "
Oct 30 13:12:02.176034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.173821    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9bd32d4c-3366-434c-8917-862060b8eda4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd32d4c-3366-434c-8917-862060b8eda4-istio-podinfo\") pod \"9bd32d4c-3366-434c-8917-862060b8eda4\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") "
Oct 30 13:12:02.176034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.174014    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd32d4c-3366-434c-8917-862060b8eda4-istio-token\" (UniqueName: \"kubernetes.io/projected/9bd32d4c-3366-434c-8917-862060b8eda4-istio-token\") pod \"9bd32d4c-3366-434c-8917-862060b8eda4\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") "
Oct 30 13:12:02.176034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.174050    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-credential-socket\") pod \"9bd32d4c-3366-434c-8917-862060b8eda4\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") "
Oct 30 13:12:02.176034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.174308    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-workload-certs\") pod \"9bd32d4c-3366-434c-8917-862060b8eda4\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") "
Oct 30 13:12:02.176034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.174757    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-workload-socket\") pod \"9bd32d4c-3366-434c-8917-862060b8eda4\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") "
Oct 30 13:12:02.176034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.174881    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd32d4c-3366-434c-8917-862060b8eda4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd32d4c-3366-434c-8917-862060b8eda4-istiod-ca-cert\") pod \"9bd32d4c-3366-434c-8917-862060b8eda4\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") "
Oct 30 13:12:02.176034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.174958    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-istio-data\") pod \"9bd32d4c-3366-434c-8917-862060b8eda4\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") "
Oct 30 13:12:02.176034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.175004    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-istio-envoy\") pod \"9bd32d4c-3366-434c-8917-862060b8eda4\" (UID: \"9bd32d4c-3366-434c-8917-862060b8eda4\") "
Oct 30 13:12:02.176034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.176021    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.182619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.176065    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.182619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.176080    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.182619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.176093    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.182619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.176107    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.182619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.176139    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r492t\" (UniqueName: \"kubernetes.io/projected/761ab5b8-a944-4b6a-acbc-327f8523ad5c-kube-api-access-r492t\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.182619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.176154    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.182619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.176167    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.182619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.176180    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761ab5b8-a944-4b6a-acbc-327f8523ad5c-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.182619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.176583    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-workload-socket" pod "9bd32d4c-3366-434c-8917-862060b8eda4" (UID: "9bd32d4c-3366-434c-8917-862060b8eda4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.182619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.177244    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-workload-certs" pod "9bd32d4c-3366-434c-8917-862060b8eda4" (UID: "9bd32d4c-3366-434c-8917-862060b8eda4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.182619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.178013    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd32d4c-3366-434c-8917-862060b8eda4-istiod-ca-cert" pod "9bd32d4c-3366-434c-8917-862060b8eda4" (UID: "9bd32d4c-3366-434c-8917-862060b8eda4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:02.182619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.179096    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-istio-data" pod "9bd32d4c-3366-434c-8917-862060b8eda4" (UID: "9bd32d4c-3366-434c-8917-862060b8eda4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.182619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.179628    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-credential-socket" pod "9bd32d4c-3366-434c-8917-862060b8eda4" (UID: "9bd32d4c-3366-434c-8917-862060b8eda4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.194788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.194345    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-istio-envoy" pod "9bd32d4c-3366-434c-8917-862060b8eda4" (UID: "9bd32d4c-3366-434c-8917-862060b8eda4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.198161 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.196907    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9bd32d4c-3366-434c-8917-862060b8eda4-istio-podinfo" pod "9bd32d4c-3366-434c-8917-862060b8eda4" (UID: "9bd32d4c-3366-434c-8917-862060b8eda4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:12:02.198161 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.197331    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd32d4c-3366-434c-8917-862060b8eda4-istio-token" pod "9bd32d4c-3366-434c-8917-862060b8eda4" (UID: "9bd32d4c-3366-434c-8917-862060b8eda4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:02.199864 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.199831    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd32d4c-3366-434c-8917-862060b8eda4-kube-api-access-8r9cj" pod "9bd32d4c-3366-434c-8917-862060b8eda4" (UID: "9bd32d4c-3366-434c-8917-862060b8eda4"). InnerVolumeSpecName "kube-api-access-8r9cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:02.278403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.278313    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.278403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.278340    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.278403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.278350    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r9cj\" (UniqueName: \"kubernetes.io/projected/9bd32d4c-3366-434c-8917-862060b8eda4-kube-api-access-8r9cj\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.278403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.278360    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9bd32d4c-3366-434c-8917-862060b8eda4-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.278403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.278368    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9bd32d4c-3366-434c-8917-862060b8eda4-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.278403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.278376    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.278403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.278384    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.278403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.278393    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9bd32d4c-3366-434c-8917-862060b8eda4-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.278403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.278402    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9bd32d4c-3366-434c-8917-862060b8eda4-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.474812 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.474785    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:12:02.583243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.583185    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-credential-socket\") pod \"40bb5401-70e8-4806-8739-ae835600784e\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") "
Oct 30 13:12:02.583504 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.583481    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-credential-socket" pod "40bb5401-70e8-4806-8739-ae835600784e" (UID: "40bb5401-70e8-4806-8739-ae835600784e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.583987 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.583961    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40bb5401-70e8-4806-8739-ae835600784e-istiod-ca-cert" pod "40bb5401-70e8-4806-8739-ae835600784e" (UID: "40bb5401-70e8-4806-8739-ae835600784e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:02.584073 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.583595    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40bb5401-70e8-4806-8739-ae835600784e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bb5401-70e8-4806-8739-ae835600784e-istiod-ca-cert\") pod \"40bb5401-70e8-4806-8739-ae835600784e\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") "
Oct 30 13:12:02.584073 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.584055    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bb5401-70e8-4806-8739-ae835600784e-kube-api-access-rstlb\" (UniqueName: \"kubernetes.io/projected/40bb5401-70e8-4806-8739-ae835600784e-kube-api-access-rstlb\") pod \"40bb5401-70e8-4806-8739-ae835600784e\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") "
Oct 30 13:12:02.584779 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.584755    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40bb5401-70e8-4806-8739-ae835600784e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bb5401-70e8-4806-8739-ae835600784e-istio-podinfo\") pod \"40bb5401-70e8-4806-8739-ae835600784e\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") "
Oct 30 13:12:02.584927 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.584910    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-istio-envoy\") pod \"40bb5401-70e8-4806-8739-ae835600784e\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") "
Oct 30 13:12:02.585505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.585477    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40bb5401-70e8-4806-8739-ae835600784e-istio-token\" (UniqueName: \"kubernetes.io/projected/40bb5401-70e8-4806-8739-ae835600784e-istio-token\") pod \"40bb5401-70e8-4806-8739-ae835600784e\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") "
Oct 30 13:12:02.585618 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.585529    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-workload-certs\") pod \"40bb5401-70e8-4806-8739-ae835600784e\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") "
Oct 30 13:12:02.585732 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.585665    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-workload-socket\") pod \"40bb5401-70e8-4806-8739-ae835600784e\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") "
Oct 30 13:12:02.585784 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.585730    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-istio-data\") pod \"40bb5401-70e8-4806-8739-ae835600784e\" (UID: \"40bb5401-70e8-4806-8739-ae835600784e\") "
Oct 30 13:12:02.586762 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.586733    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-istio-data" pod "40bb5401-70e8-4806-8739-ae835600784e" (UID: "40bb5401-70e8-4806-8739-ae835600784e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.586871 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.586763    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-workload-socket" pod "40bb5401-70e8-4806-8739-ae835600784e" (UID: "40bb5401-70e8-4806-8739-ae835600784e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.587961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.587662    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.587961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.587686    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.587961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.587699    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.587961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.587711    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40bb5401-70e8-4806-8739-ae835600784e-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.587961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.587771    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bb5401-70e8-4806-8739-ae835600784e-kube-api-access-rstlb" pod "40bb5401-70e8-4806-8739-ae835600784e" (UID: "40bb5401-70e8-4806-8739-ae835600784e"). InnerVolumeSpecName "kube-api-access-rstlb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:02.588370 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.588338    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-workload-certs" pod "40bb5401-70e8-4806-8739-ae835600784e" (UID: "40bb5401-70e8-4806-8739-ae835600784e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.589813 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.589790    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-istio-envoy" pod "40bb5401-70e8-4806-8739-ae835600784e" (UID: "40bb5401-70e8-4806-8739-ae835600784e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:12:02.589912 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.589899    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40bb5401-70e8-4806-8739-ae835600784e-istio-token" pod "40bb5401-70e8-4806-8739-ae835600784e" (UID: "40bb5401-70e8-4806-8739-ae835600784e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:02.590168 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.590150    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40bb5401-70e8-4806-8739-ae835600784e-istio-podinfo" pod "40bb5401-70e8-4806-8739-ae835600784e" (UID: "40bb5401-70e8-4806-8739-ae835600784e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:12:02.689147 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.689104    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40bb5401-70e8-4806-8739-ae835600784e-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.689147 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.689136    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.689147 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.689147    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rstlb\" (UniqueName: \"kubernetes.io/projected/40bb5401-70e8-4806-8739-ae835600784e-kube-api-access-rstlb\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.689147 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.689155    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40bb5401-70e8-4806-8739-ae835600784e-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.689534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.689163    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40bb5401-70e8-4806-8739-ae835600784e-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:02.998855 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.998825    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"
Oct 30 13:12:02.998855 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.998836    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt" event={"ID":"9bd32d4c-3366-434c-8917-862060b8eda4","Type":"ContainerDied","Data":"11451709d7176ebb47d54290aa2c1786b28af428f5e43ffcda78e041bfc863c2"}
Oct 30 13:12:02.999363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:02.998899    5893 scope.go:122] "RemoveContainer" containerID="85718abfee2812b0dd3dc4d45c870f81e4dbecbae2c6e210d5e87baae29d2f72"
Oct 30 13:12:03.000715 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:03.000696    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"
Oct 30 13:12:03.000840 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:03.000720    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"
Oct 30 13:12:03.000840 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:03.000711    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp" event={"ID":"40bb5401-70e8-4806-8739-ae835600784e","Type":"ContainerDied","Data":"a2fae7803b1d8ce0e1263223da7ed74d7065a7dc4f40535cf836c0db30f8f6fb"}
Oct 30 13:12:03.010673 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:03.010656    5893 scope.go:122] "RemoveContainer" containerID="e7d936761dc30e6214efa57d8c9cc98ccd4341ab55966e195d51b743a59b516f"
Oct 30 13:12:03.056456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:03.056420    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"]
Oct 30 13:12:03.061862 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:03.061831    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-zf4jt"]
Oct 30 13:12:03.099002 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:03.098973    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"]
Oct 30 13:12:03.103480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:03.103458    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-fvxqv"]
Oct 30 13:12:03.136479 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:03.136454    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"]
Oct 30 13:12:03.141107 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:03.141082    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-jwxnp"]
Oct 30 13:12:04.532998 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:04.532965    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40bb5401-70e8-4806-8739-ae835600784e" path="/var/lib/kubelet/pods/40bb5401-70e8-4806-8739-ae835600784e/volumes"
Oct 30 13:12:04.533484 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:04.533469    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="761ab5b8-a944-4b6a-acbc-327f8523ad5c" path="/var/lib/kubelet/pods/761ab5b8-a944-4b6a-acbc-327f8523ad5c/volumes"
Oct 30 13:12:04.533922 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:04.533909    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd32d4c-3366-434c-8917-862060b8eda4" path="/var/lib/kubelet/pods/9bd32d4c-3366-434c-8917-862060b8eda4/volumes"
Oct 30 13:12:08.099612 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.099579    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"]
Oct 30 13:12:08.100561 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.100540    5893 cpu_manager.go:414] "containerMap: removing container" podUID="761ab5b8-a944-4b6a-acbc-327f8523ad5c" containerName="istio-proxy"
Oct 30 13:12:08.100694 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.100562    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="761ab5b8-a944-4b6a-acbc-327f8523ad5c" containerName="istio-proxy"
Oct 30 13:12:08.100694 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.100587    5893 cpu_manager.go:414] "containerMap: removing container" podUID="9bd32d4c-3366-434c-8917-862060b8eda4" containerName="istio-proxy"
Oct 30 13:12:08.100694 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.100594    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd32d4c-3366-434c-8917-862060b8eda4" containerName="istio-proxy"
Oct 30 13:12:08.100694 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.100603    5893 cpu_manager.go:414] "containerMap: removing container" podUID="40bb5401-70e8-4806-8739-ae835600784e" containerName="istio-proxy"
Oct 30 13:12:08.100694 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.100609    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40bb5401-70e8-4806-8739-ae835600784e" containerName="istio-proxy"
Oct 30 13:12:08.100932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.100819    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="761ab5b8-a944-4b6a-acbc-327f8523ad5c" containerName="istio-proxy"
Oct 30 13:12:08.100932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.100836    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd32d4c-3366-434c-8917-862060b8eda4" containerName="istio-proxy"
Oct 30 13:12:08.100932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.100847    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="40bb5401-70e8-4806-8739-ae835600784e" containerName="istio-proxy"
Oct 30 13:12:08.106307 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.106261    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:08.123291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.123252    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"]
Oct 30 13:12:08.130199 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.130161    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1c0e6c42-serving-cert\""
Oct 30 13:12:08.196619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.196584    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fb9ed96-44c3-449d-90ea-055bd127799f-maas-api-tls\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:08.196814 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.196659    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb9ed96-44c3-449d-90ea-055bd127799f-openshift-service-ca\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:08.196814 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.196794    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fb9ed96-44c3-449d-90ea-055bd127799f-metrics-certs\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:08.196941 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.196861    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jqgc\" (UniqueName: \"kubernetes.io/projected/9fb9ed96-44c3-449d-90ea-055bd127799f-kube-api-access-5jqgc\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:08.298610 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.298579    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fb9ed96-44c3-449d-90ea-055bd127799f-maas-api-tls\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:08.298804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.298648    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb9ed96-44c3-449d-90ea-055bd127799f-openshift-service-ca\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:08.298804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.298693    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fb9ed96-44c3-449d-90ea-055bd127799f-metrics-certs\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:08.298804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.298736    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jqgc\" (UniqueName: \"kubernetes.io/projected/9fb9ed96-44c3-449d-90ea-055bd127799f-kube-api-access-5jqgc\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:08.299465 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.299438    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb9ed96-44c3-449d-90ea-055bd127799f-openshift-service-ca\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:08.301310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.301288    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fb9ed96-44c3-449d-90ea-055bd127799f-maas-api-tls\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:08.301383 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.301341    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fb9ed96-44c3-449d-90ea-055bd127799f-metrics-certs\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:08.350892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.350818    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jqgc\" (UniqueName: \"kubernetes.io/projected/9fb9ed96-44c3-449d-90ea-055bd127799f-kube-api-access-5jqgc\") pod \"maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:08.426941 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.426912    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:08.781816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:08.781768    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"]
Oct 30 13:12:08.782721 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:12:08.782688    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb9ed96_44c3_449d_90ea_055bd127799f.slice/crio-557e584a6c18299abc27de8948c7abea17a3bf20b3b365600dbb17d1e9f8f1d2 WatchSource:0}: Error finding container 557e584a6c18299abc27de8948c7abea17a3bf20b3b365600dbb17d1e9f8f1d2: Status 404 returned error can't find the container with id 557e584a6c18299abc27de8948c7abea17a3bf20b3b365600dbb17d1e9f8f1d2
Oct 30 13:12:09.083702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:09.083615    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd" event={"ID":"9fb9ed96-44c3-449d-90ea-055bd127799f","Type":"ContainerStarted","Data":"557e584a6c18299abc27de8948c7abea17a3bf20b3b365600dbb17d1e9f8f1d2"}
Oct 30 13:12:10.096209 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:10.096168    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd" event={"ID":"9fb9ed96-44c3-449d-90ea-055bd127799f","Type":"ContainerStarted","Data":"dc2197c9975e75b519a738f1454992c0eaac601b933a267e31cec294ad14f55a"}
Oct 30 13:12:10.096675 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:10.096286    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:10.143625 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:10.143565    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd" podStartSLOduration=1.817071477 podStartE2EDuration="2.143544806s" totalImagesPullingTime="326.473329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:10.138648414 +0000 UTC m=+1571.722078117" watchObservedRunningTime="2026-10-30 13:12:10.143544806 +0000 UTC m=+1571.726974501"
Oct 30 13:12:16.110988 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:16.110955    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:12:17.225686 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.225652    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"]
Oct 30 13:12:17.230312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.230295    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:17.260303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.258850    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-abbc6c-serving-cert\""
Oct 30 13:12:17.278057 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.277990    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"]
Oct 30 13:12:17.338911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.338871    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44d11615-8178-4efa-9468-ce4a6029e6db-metrics-certs\") pod \"maas-api-e2e-api-a-abbc6c-664db4b855-dthxr\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:17.339081 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.338924    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d11615-8178-4efa-9468-ce4a6029e6db-openshift-service-ca\") pod \"maas-api-e2e-api-a-abbc6c-664db4b855-dthxr\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:17.339081 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.339048    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cvgw\" (UniqueName: \"kubernetes.io/projected/44d11615-8178-4efa-9468-ce4a6029e6db-kube-api-access-2cvgw\") pod \"maas-api-e2e-api-a-abbc6c-664db4b855-dthxr\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:17.339215 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.339203    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44d11615-8178-4efa-9468-ce4a6029e6db-maas-api-tls\") pod \"maas-api-e2e-api-a-abbc6c-664db4b855-dthxr\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:17.443426 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.443392    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d11615-8178-4efa-9468-ce4a6029e6db-openshift-service-ca\") pod \"maas-api-e2e-api-a-abbc6c-664db4b855-dthxr\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:17.443804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.442137    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d11615-8178-4efa-9468-ce4a6029e6db-openshift-service-ca\") pod \"maas-api-e2e-api-a-abbc6c-664db4b855-dthxr\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:17.458659 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.457386    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cvgw\" (UniqueName: \"kubernetes.io/projected/44d11615-8178-4efa-9468-ce4a6029e6db-kube-api-access-2cvgw\") pod \"maas-api-e2e-api-a-abbc6c-664db4b855-dthxr\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:17.458659 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.457693    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44d11615-8178-4efa-9468-ce4a6029e6db-maas-api-tls\") pod \"maas-api-e2e-api-a-abbc6c-664db4b855-dthxr\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:17.458659 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.457882    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44d11615-8178-4efa-9468-ce4a6029e6db-metrics-certs\") pod \"maas-api-e2e-api-a-abbc6c-664db4b855-dthxr\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:17.470868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.470836    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44d11615-8178-4efa-9468-ce4a6029e6db-metrics-certs\") pod \"maas-api-e2e-api-a-abbc6c-664db4b855-dthxr\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:17.471874 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.471835    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44d11615-8178-4efa-9468-ce4a6029e6db-maas-api-tls\") pod \"maas-api-e2e-api-a-abbc6c-664db4b855-dthxr\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:17.482250 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.482158    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cvgw\" (UniqueName: \"kubernetes.io/projected/44d11615-8178-4efa-9468-ce4a6029e6db-kube-api-access-2cvgw\") pod \"maas-api-e2e-api-a-abbc6c-664db4b855-dthxr\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:17.576641 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.576602    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:17.902714 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:17.902692    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"]
Oct 30 13:12:17.903943 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:12:17.903920    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d11615_8178_4efa_9468_ce4a6029e6db.slice/crio-9fd53dee814a901891d5c90cbdba76d304ca860850a1d786451baa599e57a8a3 WatchSource:0}: Error finding container 9fd53dee814a901891d5c90cbdba76d304ca860850a1d786451baa599e57a8a3: Status 404 returned error can't find the container with id 9fd53dee814a901891d5c90cbdba76d304ca860850a1d786451baa599e57a8a3
Oct 30 13:12:18.157449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:18.157359    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr" event={"ID":"44d11615-8178-4efa-9468-ce4a6029e6db","Type":"ContainerStarted","Data":"9fd53dee814a901891d5c90cbdba76d304ca860850a1d786451baa599e57a8a3"}
Oct 30 13:12:18.649029 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:12:18.646936    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:12:19.169620 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:19.169581    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr" event={"ID":"44d11615-8178-4efa-9468-ce4a6029e6db","Type":"ContainerStarted","Data":"ef02ebe5ee8a21251c343f7c9e29a9ac9bd1e8b9d490efe20a12e044447d3973"}
Oct 30 13:12:19.169811 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:19.169677    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:19.214707 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:19.214659    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr" podStartSLOduration=1.847287923 podStartE2EDuration="2.214644904s" totalImagesPullingTime="367.356981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:19.211412756 +0000 UTC m=+1580.794842448" watchObservedRunningTime="2026-10-30 13:12:19.214644904 +0000 UTC m=+1580.798074597"
Oct 30 13:12:22.669922 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:22.669880    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"]
Oct 30 13:12:25.193748 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:25.193713    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:12:27.670352 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:27.670313    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn" podUID="9395056f-a35b-457d-9c77-3877c5571e39" containerName="payload-pre-processing" containerID="cri-o://d22b05b0689bea4202fd013086ac944009786b90c83f53a1f4e2752f1aa7a7f8" gracePeriod=25
Oct 30 13:12:27.971370 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:27.971344    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"
Oct 30 13:12:28.090757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.090720    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9395056f-a35b-457d-9c77-3877c5571e39-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9395056f-a35b-457d-9c77-3877c5571e39-plugins-config-volume\") pod \"9395056f-a35b-457d-9c77-3877c5571e39\" (UID: \"9395056f-a35b-457d-9c77-3877c5571e39\") "
Oct 30 13:12:28.090918 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.090824    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9395056f-a35b-457d-9c77-3877c5571e39-kube-api-access-dsv42\" (UniqueName: \"kubernetes.io/projected/9395056f-a35b-457d-9c77-3877c5571e39-kube-api-access-dsv42\") pod \"9395056f-a35b-457d-9c77-3877c5571e39\" (UID: \"9395056f-a35b-457d-9c77-3877c5571e39\") "
Oct 30 13:12:28.091198 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.091165    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9395056f-a35b-457d-9c77-3877c5571e39-plugins-config-volume" pod "9395056f-a35b-457d-9c77-3877c5571e39" (UID: "9395056f-a35b-457d-9c77-3877c5571e39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:28.091748 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.091725    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9395056f-a35b-457d-9c77-3877c5571e39-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:28.093140 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.093110    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9395056f-a35b-457d-9c77-3877c5571e39-kube-api-access-dsv42" pod "9395056f-a35b-457d-9c77-3877c5571e39" (UID: "9395056f-a35b-457d-9c77-3877c5571e39"). InnerVolumeSpecName "kube-api-access-dsv42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:28.193250 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.193168    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsv42\" (UniqueName: \"kubernetes.io/projected/9395056f-a35b-457d-9c77-3877c5571e39-kube-api-access-dsv42\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:12:28.255230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.255194    5893 generic.go:356] "Generic (PLEG): container finished" podID="9395056f-a35b-457d-9c77-3877c5571e39" containerID="d22b05b0689bea4202fd013086ac944009786b90c83f53a1f4e2752f1aa7a7f8" exitCode=0
Oct 30 13:12:28.255431 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.255249    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"
Oct 30 13:12:28.255431 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.255320    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn" event={"ID":"9395056f-a35b-457d-9c77-3877c5571e39","Type":"ContainerDied","Data":"d22b05b0689bea4202fd013086ac944009786b90c83f53a1f4e2752f1aa7a7f8"}
Oct 30 13:12:28.255431 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.255390    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn" event={"ID":"9395056f-a35b-457d-9c77-3877c5571e39","Type":"ContainerDied","Data":"24a177731affc25bed46bb98d0a048da3badd0403cd31a5f912fc689e740caff"}
Oct 30 13:12:28.255431 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.255421    5893 scope.go:122] "RemoveContainer" containerID="d22b05b0689bea4202fd013086ac944009786b90c83f53a1f4e2752f1aa7a7f8"
Oct 30 13:12:28.267098 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.267078    5893 scope.go:122] "RemoveContainer" containerID="d22b05b0689bea4202fd013086ac944009786b90c83f53a1f4e2752f1aa7a7f8"
Oct 30 13:12:28.267427 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:12:28.267408    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d22b05b0689bea4202fd013086ac944009786b90c83f53a1f4e2752f1aa7a7f8\": container with ID starting with d22b05b0689bea4202fd013086ac944009786b90c83f53a1f4e2752f1aa7a7f8 not found: ID does not exist" containerID="d22b05b0689bea4202fd013086ac944009786b90c83f53a1f4e2752f1aa7a7f8"
Oct 30 13:12:28.267482 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.267436    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d22b05b0689bea4202fd013086ac944009786b90c83f53a1f4e2752f1aa7a7f8"} err="failed to get container status \"d22b05b0689bea4202fd013086ac944009786b90c83f53a1f4e2752f1aa7a7f8\": rpc error: code = NotFound desc = could not find container \"d22b05b0689bea4202fd013086ac944009786b90c83f53a1f4e2752f1aa7a7f8\": container with ID starting with d22b05b0689bea4202fd013086ac944009786b90c83f53a1f4e2752f1aa7a7f8 not found: ID does not exist"
Oct 30 13:12:28.310885 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.310850    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"]
Oct 30 13:12:28.315320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.315288    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75abce-dfb9f7f55-tmxjn"]
Oct 30 13:12:28.533974 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:28.533942    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9395056f-a35b-457d-9c77-3877c5571e39" path="/var/lib/kubelet/pods/9395056f-a35b-457d-9c77-3877c5571e39/volumes"
Oct 30 13:12:30.229115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.229084    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"]
Oct 30 13:12:30.229889 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.229874    5893 cpu_manager.go:414] "containerMap: removing container" podUID="9395056f-a35b-457d-9c77-3877c5571e39" containerName="payload-pre-processing"
Oct 30 13:12:30.229940 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.229891    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9395056f-a35b-457d-9c77-3877c5571e39" containerName="payload-pre-processing"
Oct 30 13:12:30.230041 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.230030    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="9395056f-a35b-457d-9c77-3877c5571e39" containerName="payload-pre-processing"
Oct 30 13:12:30.234993 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.234974    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:30.260952 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.260917    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"]
Oct 30 13:12:30.268994 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.268965    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-716b4c-serving-cert\""
Oct 30 13:12:30.328615 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.328584    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rt4t\" (UniqueName: \"kubernetes.io/projected/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-kube-api-access-6rt4t\") pod \"maas-api-e2e-api-b-716b4c-748fc76498-r9rsz\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:30.328892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.328870    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-maas-api-tls\") pod \"maas-api-e2e-api-b-716b4c-748fc76498-r9rsz\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:30.329009 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.328990    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-metrics-certs\") pod \"maas-api-e2e-api-b-716b4c-748fc76498-r9rsz\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:30.329081 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.329065    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-openshift-service-ca\") pod \"maas-api-e2e-api-b-716b4c-748fc76498-r9rsz\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:30.430293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.430239    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-maas-api-tls\") pod \"maas-api-e2e-api-b-716b4c-748fc76498-r9rsz\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:30.430485 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.430377    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-metrics-certs\") pod \"maas-api-e2e-api-b-716b4c-748fc76498-r9rsz\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:30.430485 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.430453    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-openshift-service-ca\") pod \"maas-api-e2e-api-b-716b4c-748fc76498-r9rsz\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:30.430608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.430556    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rt4t\" (UniqueName: \"kubernetes.io/projected/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-kube-api-access-6rt4t\") pod \"maas-api-e2e-api-b-716b4c-748fc76498-r9rsz\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:30.433154 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.433123    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-openshift-service-ca\") pod \"maas-api-e2e-api-b-716b4c-748fc76498-r9rsz\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:30.437738 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.434515    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-maas-api-tls\") pod \"maas-api-e2e-api-b-716b4c-748fc76498-r9rsz\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:30.438841 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.438818    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-metrics-certs\") pod \"maas-api-e2e-api-b-716b4c-748fc76498-r9rsz\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:30.456098 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.456067    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rt4t\" (UniqueName: \"kubernetes.io/projected/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-kube-api-access-6rt4t\") pod \"maas-api-e2e-api-b-716b4c-748fc76498-r9rsz\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:30.550130 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.550036    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:30.869878 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:30.869853    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"]
Oct 30 13:12:30.871644 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:12:30.871607    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57facfc2_95f0_41d0_ab18_1b60dab9bc9d.slice/crio-d499a888fda72a636c520d23e8a0e86b71955927dd83f3cdbfd0567682913943 WatchSource:0}: Error finding container d499a888fda72a636c520d23e8a0e86b71955927dd83f3cdbfd0567682913943: Status 404 returned error can't find the container with id d499a888fda72a636c520d23e8a0e86b71955927dd83f3cdbfd0567682913943
Oct 30 13:12:31.291565 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:31.291517    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz" event={"ID":"57facfc2-95f0-41d0-ab18-1b60dab9bc9d","Type":"ContainerStarted","Data":"5ab735c333881903ed196c7f9137569e2d4be3f79b556f6211aa2de2969ee76a"}
Oct 30 13:12:31.292037 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:31.291597    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:31.292037 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:31.291623    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz" event={"ID":"57facfc2-95f0-41d0-ab18-1b60dab9bc9d","Type":"ContainerStarted","Data":"d499a888fda72a636c520d23e8a0e86b71955927dd83f3cdbfd0567682913943"}
Oct 30 13:12:31.335968 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:31.335908    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz" podStartSLOduration=1.006062968 podStartE2EDuration="1.335894514s" totalImagesPullingTime="329.831546ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:31.334526227 +0000 UTC m=+1592.917955911" watchObservedRunningTime="2026-10-30 13:12:31.335894514 +0000 UTC m=+1592.919324206"
Oct 30 13:12:37.304165 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:37.304134    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:12:38.474873 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:38.474834    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"]
Oct 30 13:12:38.479977 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:38.479947    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"
Oct 30 13:12:38.504894 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:38.504865    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"]
Oct 30 13:12:38.511337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:38.511312    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-2a3e01\""
Oct 30 13:12:38.583950 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:38.583910    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27960fcb-f56f-42e8-8233-430811c533e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw\" (UID: \"27960fcb-f56f-42e8-8233-430811c533e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"
Oct 30 13:12:38.584580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:38.584555    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc9vm\" (UniqueName: \"kubernetes.io/projected/27960fcb-f56f-42e8-8233-430811c533e8-kube-api-access-kc9vm\") pod \"payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw\" (UID: \"27960fcb-f56f-42e8-8233-430811c533e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"
Oct 30 13:12:38.715301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:38.714140    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27960fcb-f56f-42e8-8233-430811c533e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw\" (UID: \"27960fcb-f56f-42e8-8233-430811c533e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"
Oct 30 13:12:38.715301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:38.714237    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc9vm\" (UniqueName: \"kubernetes.io/projected/27960fcb-f56f-42e8-8233-430811c533e8-kube-api-access-kc9vm\") pod \"payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw\" (UID: \"27960fcb-f56f-42e8-8233-430811c533e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"
Oct 30 13:12:38.715301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:38.715173    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27960fcb-f56f-42e8-8233-430811c533e8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw\" (UID: \"27960fcb-f56f-42e8-8233-430811c533e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"
Oct 30 13:12:38.741359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:38.741157    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc9vm\" (UniqueName: \"kubernetes.io/projected/27960fcb-f56f-42e8-8233-430811c533e8-kube-api-access-kc9vm\") pod \"payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw\" (UID: \"27960fcb-f56f-42e8-8233-430811c533e8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"
Oct 30 13:12:38.802296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:38.801398    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"
Oct 30 13:12:39.137361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:39.137333    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"]
Oct 30 13:12:39.138861 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:12:39.138837    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27960fcb_f56f_42e8_8233_430811c533e8.slice/crio-df5b64a87de5ab0788550d2f2ece4e547989752f53ec4136a359e38335ee1f75 WatchSource:0}: Error finding container df5b64a87de5ab0788550d2f2ece4e547989752f53ec4136a359e38335ee1f75: Status 404 returned error can't find the container with id df5b64a87de5ab0788550d2f2ece4e547989752f53ec4136a359e38335ee1f75
Oct 30 13:12:39.364783 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:39.364748    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw" event={"ID":"27960fcb-f56f-42e8-8233-430811c533e8","Type":"ContainerStarted","Data":"14e70bc12788b902dd361909bd24ce61462975a653bd2a0078adf971f7c9d0ac"}
Oct 30 13:12:39.364989 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:39.364838    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw" event={"ID":"27960fcb-f56f-42e8-8233-430811c533e8","Type":"ContainerStarted","Data":"df5b64a87de5ab0788550d2f2ece4e547989752f53ec4136a359e38335ee1f75"}
Oct 30 13:12:39.365248 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:39.364978    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"
Oct 30 13:12:39.412727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:39.412101    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw" podStartSLOduration=1.412074443 podStartE2EDuration="1.412074443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:39.405406216 +0000 UTC m=+1600.988835934" watchObservedRunningTime="2026-10-30 13:12:39.412074443 +0000 UTC m=+1600.995504173"
Oct 30 13:12:48.648479 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:12:48.648443    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:12:50.025340 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.022674    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"]
Oct 30 13:12:50.028871 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.028833    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.048357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.048330    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"]
Oct 30 13:12:50.060233 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.060204    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-14e8a4-openshift-default-dockercfg-wnk87\""
Oct 30 13:12:50.125333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.125296    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-credential-socket\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.125523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.125348    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb69n\" (UniqueName: \"kubernetes.io/projected/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-kube-api-access-pb69n\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.125575 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.125509    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-envoy\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.125640 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.125582    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-workload-socket\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.125698 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.125668    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-workload-certs\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.125754 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.125705    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-data\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.125801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.125780    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-podinfo\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.125868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.125851    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-token\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.126033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.126016    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istiod-ca-cert\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.231297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.230381    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istiod-ca-cert\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.231297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.230545    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-credential-socket\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.231297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.230614    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb69n\" (UniqueName: \"kubernetes.io/projected/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-kube-api-access-pb69n\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.231297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.230734    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-envoy\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.231297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.230798    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-workload-socket\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.231297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.230835    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-workload-certs\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.231297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.230860    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-data\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.231297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.230884    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-podinfo\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.231297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.230985    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-token\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.233304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.232128    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istiod-ca-cert\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.234521 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.234497    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-credential-socket\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.238975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.238951    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-workload-certs\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.239530 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.239507    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-workload-socket\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.239963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.239943    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-data\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.245227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.245171    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-podinfo\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.277449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.268017    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-envoy\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.289229 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.287226    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-token\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.290168 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.290133    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb69n\" (UniqueName: \"kubernetes.io/projected/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-kube-api-access-pb69n\") pod \"e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") " pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.357871 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.356664    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:50.412369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.408210    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"
Oct 30 13:12:50.789308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.782364    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"]
Oct 30 13:12:50.924218 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.923256    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:50.924218 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.923384    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:50.924218 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:50.923440    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:12:51.711563 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:51.711470    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng" event={"ID":"d716f5ae-5dba-41f1-aeef-1d4fdeadba33","Type":"ContainerStarted","Data":"b24d5ad44f41475ff3a63937703f9b9a9ef715b71191ee3c7871b5d530f0d50c"}
Oct 30 13:12:51.844305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:51.840491    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:51.862402 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:51.862360    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:52.562300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:52.560411    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:52.585576 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:52.585538    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:52.615366 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:52.615130    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:52.749471 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:52.749432    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng" event={"ID":"d716f5ae-5dba-41f1-aeef-1d4fdeadba33","Type":"ContainerStarted","Data":"207ecee35167d3224270e421a47d925b1f95488a9ecd3ebc0cc9e037e32f2b68"}
Oct 30 13:12:52.764420 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:52.764366    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Oct 30 13:12:52.771499 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:52.771465    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:52.786825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:52.777796    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:52.845672 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:52.845546    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:52.849660 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:52.849634    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:12:52.873618 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:52.873552    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng" podStartSLOduration=3.8735292059999997 podStartE2EDuration="3.873529206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:52.821207487 +0000 UTC m=+1614.404637205" watchObservedRunningTime="2026-10-30 13:12:52.873529206 +0000 UTC m=+1614.456958900"
Oct 30 13:12:52.898595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:52.898549    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:12:52.909432 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:52.905181    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:12:53.370361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:53.358687    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:53.411123 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:53.409891    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:53.604300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:53.588491    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:53.604300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:53.589254    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:12:53.714393 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:53.696375    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:53.714393 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:53.697370    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:12:53.824301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:53.782457    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Oct 30 13:12:53.854317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:53.854267    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:12:53.887744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:53.878392    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:53.921404 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:53.921355    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded"
Oct 30 13:12:54.152387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:54.136460    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:54.152387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:54.140557    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:12:54.562301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:54.562006    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:54.603311 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:54.602770    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:54.739300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:54.730619    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:12:54.891365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:54.840908    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:12:54.891365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:54.840992    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:12:55.010302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:55.009730    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:12:55.124296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:55.109089    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:12:55.212573 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:55.211915    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:12:55.229466 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:55.229439    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:12:57.934210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:57.934155    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"]
Oct 30 13:12:57.969366 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:57.968149    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"
Oct 30 13:12:58.023911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:58.014708    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-14e8a4\""
Oct 30 13:12:58.023911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:58.021077    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"]
Oct 30 13:12:58.023911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:58.022876    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1425d7d5-e09a-4a70-98c9-58c50474a2da-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn\" (UID: \"1425d7d5-e09a-4a70-98c9-58c50474a2da\") " pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"
Oct 30 13:12:58.023911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:58.023077    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b22l6\" (UniqueName: \"kubernetes.io/projected/1425d7d5-e09a-4a70-98c9-58c50474a2da-kube-api-access-b22l6\") pod \"payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn\" (UID: \"1425d7d5-e09a-4a70-98c9-58c50474a2da\") " pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"
Oct 30 13:12:58.027993 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:58.027964    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-14e8a4-dockercfg-wv67l\""
Oct 30 13:12:58.156351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:58.141972    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b22l6\" (UniqueName: \"kubernetes.io/projected/1425d7d5-e09a-4a70-98c9-58c50474a2da-kube-api-access-b22l6\") pod \"payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn\" (UID: \"1425d7d5-e09a-4a70-98c9-58c50474a2da\") " pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"
Oct 30 13:12:58.156351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:58.142342    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1425d7d5-e09a-4a70-98c9-58c50474a2da-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn\" (UID: \"1425d7d5-e09a-4a70-98c9-58c50474a2da\") " pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"
Oct 30 13:12:58.219639 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:58.219513    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1425d7d5-e09a-4a70-98c9-58c50474a2da-plugins-config-volume\") pod \"payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn\" (UID: \"1425d7d5-e09a-4a70-98c9-58c50474a2da\") " pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"
Oct 30 13:12:58.326103 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:58.321113    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b22l6\" (UniqueName: \"kubernetes.io/projected/1425d7d5-e09a-4a70-98c9-58c50474a2da-kube-api-access-b22l6\") pod \"payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn\" (UID: \"1425d7d5-e09a-4a70-98c9-58c50474a2da\") " pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"
Oct 30 13:12:58.666120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:58.666034    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-14e8a4-dockercfg-wv67l\""
Oct 30 13:12:58.678103 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:58.676518    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"
Oct 30 13:12:59.069557 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:59.069529    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"]
Oct 30 13:12:59.337688 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:59.337552    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn" event={"ID":"1425d7d5-e09a-4a70-98c9-58c50474a2da","Type":"ContainerStarted","Data":"c95a4331cb5d47548f6de82b718572b26f8831c786f9b54a3f5ddf99fdbfaba6"}
Oct 30 13:12:59.337688 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:59.337623    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn" event={"ID":"1425d7d5-e09a-4a70-98c9-58c50474a2da","Type":"ContainerStarted","Data":"1a8b406d6f4571dd505ac2df33a5e579f718521267e3aaabfc2d43f3ef266c7a"}
Oct 30 13:12:59.337688 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:59.337653    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"
Oct 30 13:12:59.380257 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:59.380199    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn" podStartSLOduration=2.3801800650000002 podStartE2EDuration="2.380180065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:59.37803581 +0000 UTC m=+1620.961465504" watchObservedRunningTime="2026-10-30 13:12:59.380180065 +0000 UTC m=+1620.963609759"
Oct 30 13:12:59.428129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:12:59.428093    5893 scope.go:122] "RemoveContainer" containerID="ab35bb80c9f018540e860f7b0ad59fba29fe679e9cf8f718e12a5729e6b4e222"
Oct 30 13:13:03.471487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:03.471222    5893 patch_prober.go:29] interesting pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": read tcp 10.134.0.2:49690->10.134.0.92:15021: read: connection reset by peer" start-of-body=
Oct 30 13:13:03.471487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:03.471342    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": read tcp 10.134.0.2:49690->10.134.0.92:15021: read: connection reset by peer"
Oct 30 13:13:04.389466 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:04.389428    5893 generic.go:356] "Generic (PLEG): container finished" podID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerID="23e2df560a06d0096ca32c0be19a7dd026b7b03e11c01a4967e34d44c277ac02" exitCode=0
Oct 30 13:13:04.389690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:04.389506    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" event={"ID":"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a","Type":"ContainerDied","Data":"23e2df560a06d0096ca32c0be19a7dd026b7b03e11c01a4967e34d44c277ac02"}
Oct 30 13:13:04.390102 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:04.390066    5893 scope.go:122] "RemoveContainer" containerID="23e2df560a06d0096ca32c0be19a7dd026b7b03e11c01a4967e34d44c277ac02"
Oct 30 13:13:04.391505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:04.391485    5893 generic.go:356] "Generic (PLEG): container finished" podID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerID="0074433727b7e1099de818eb07ed8ab63cce1d10de7fdb2641bc9fc90a1b1b70" exitCode=0
Oct 30 13:13:04.391631 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:04.391565    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" event={"ID":"2ed508c1-ee5b-47e1-b335-7453b53a6f9d","Type":"ContainerDied","Data":"0074433727b7e1099de818eb07ed8ab63cce1d10de7fdb2641bc9fc90a1b1b70"}
Oct 30 13:13:04.391995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:04.391978    5893 scope.go:122] "RemoveContainer" containerID="0074433727b7e1099de818eb07ed8ab63cce1d10de7fdb2641bc9fc90a1b1b70"
Oct 30 13:13:04.392155 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:04.392126    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:13:04.392213 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:04.392197    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:13:04.392259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:04.392230    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:13:04.394026 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:04.393998    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:13:04.394106 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:04.394081    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:13:04.394148 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:04.394129    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:13:05.402197 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:05.402159    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" event={"ID":"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a","Type":"ContainerStarted","Data":"82304be2cf824b2c0d13453744b2349d753d0ac928660a9e7563007bc631ca41"}
Oct 30 13:13:05.404064 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:05.404038    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" event={"ID":"2ed508c1-ee5b-47e1-b335-7453b53a6f9d","Type":"ContainerStarted","Data":"28bb9f875cd3d9dfcca5db15ccecd51caccadc70a15cca12b39b28ab78c4ba82"}
Oct 30 13:13:05.432642 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:05.432619    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:13:05.432902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:05.432880    5893 patch_prober.go:29] interesting pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:13:05.433010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:05.432928    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 13:13:05.916081 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:05.916043    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:13:05.916430 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:05.916398    5893 patch_prober.go:29] interesting pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:13:05.916491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:05.916459    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 13:13:06.432167 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:06.432133    5893 patch_prober.go:29] interesting pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:13:06.432581 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:06.432197    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 13:13:06.924347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:06.922659    5893 patch_prober.go:29] interesting pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:13:06.924347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:06.922715    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 13:13:07.432185 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:07.432159    5893 patch_prober.go:29] interesting pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:13:07.432671 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:07.432650    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 13:13:07.915709 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:07.915669    5893 patch_prober.go:29] interesting pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:13:07.915932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:07.915766    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 13:13:08.432476 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:08.432442    5893 patch_prober.go:29] interesting pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:13:08.432871 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:08.432503    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 13:13:08.916159 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:08.916124    5893 patch_prober.go:29] interesting pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:13:08.916363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:08.916170    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 13:13:09.432880 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:09.432847    5893 patch_prober.go:29] interesting pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:13:09.433329 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:09.432900    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 13:13:09.915859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:09.915820    5893 patch_prober.go:29] interesting pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:13:09.916073 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:09.915894    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 13:13:10.388687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:10.388656    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"
Oct 30 13:13:10.433491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:10.432795    5893 patch_prober.go:29] interesting pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Oct 30 13:13:10.433491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:10.432881    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Oct 30 13:13:10.915963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:10.915928    5893 patch_prober.go:29] interesting pod/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Oct 30 13:13:10.916156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:10.915979    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Oct 30 13:13:11.437490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:11.437454    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:13:11.438013 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:11.437997    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:13:11.438827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:11.438806    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:13:11.936215 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:11.936185    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:13:11.938789 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:11.938763    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:13:11.942047 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:11.942025    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:13:18.162518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:18.162482    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"]
Oct 30 13:13:18.167374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:18.167352    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"
Oct 30 13:13:18.199501 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:18.199470    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-11cfa6\""
Oct 30 13:13:18.201232 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:18.199818    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-11cfa6-dockercfg-hxdv9\""
Oct 30 13:13:18.212020 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:18.209480    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"]
Oct 30 13:13:18.421410 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:18.420668    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nqbt\" (UniqueName: \"kubernetes.io/projected/7e10009d-973a-440c-aa8f-89aa1ce91755-kube-api-access-7nqbt\") pod \"payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk\" (UID: \"7e10009d-973a-440c-aa8f-89aa1ce91755\") " pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"
Oct 30 13:13:18.421410 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:18.420801    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e10009d-973a-440c-aa8f-89aa1ce91755-plugins-config-volume\") pod \"payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk\" (UID: \"7e10009d-973a-440c-aa8f-89aa1ce91755\") " pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"
Oct 30 13:13:18.531296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:18.524727    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e10009d-973a-440c-aa8f-89aa1ce91755-plugins-config-volume\") pod \"payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk\" (UID: \"7e10009d-973a-440c-aa8f-89aa1ce91755\") " pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"
Oct 30 13:13:18.531296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:18.525162    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nqbt\" (UniqueName: \"kubernetes.io/projected/7e10009d-973a-440c-aa8f-89aa1ce91755-kube-api-access-7nqbt\") pod \"payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk\" (UID: \"7e10009d-973a-440c-aa8f-89aa1ce91755\") " pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"
Oct 30 13:13:18.531296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:18.526134    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e10009d-973a-440c-aa8f-89aa1ce91755-plugins-config-volume\") pod \"payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk\" (UID: \"7e10009d-973a-440c-aa8f-89aa1ce91755\") " pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"
Oct 30 13:13:18.553720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:18.553685    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nqbt\" (UniqueName: \"kubernetes.io/projected/7e10009d-973a-440c-aa8f-89aa1ce91755-kube-api-access-7nqbt\") pod \"payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk\" (UID: \"7e10009d-973a-440c-aa8f-89aa1ce91755\") " pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"
Oct 30 13:13:18.647336 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:13:18.647297    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:13:18.785893 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:18.785640    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"
Oct 30 13:13:19.115164 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:19.115131    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"]
Oct 30 13:13:19.118446 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:13:19.118415    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e10009d_973a_440c_aa8f_89aa1ce91755.slice/crio-e70e6642f8d0d10fb4abd070d2b2b464e292b4c9ad9b445ad268ef5c632ec5c2 WatchSource:0}: Error finding container e70e6642f8d0d10fb4abd070d2b2b464e292b4c9ad9b445ad268ef5c632ec5c2: Status 404 returned error can't find the container with id e70e6642f8d0d10fb4abd070d2b2b464e292b4c9ad9b445ad268ef5c632ec5c2
Oct 30 13:13:19.621127 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:19.621093    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk" event={"ID":"7e10009d-973a-440c-aa8f-89aa1ce91755","Type":"ContainerStarted","Data":"4a3ea66edacdb048e09e77eef40ea7778d9789625c289b5e13704ad2dd49bba3"}
Oct 30 13:13:19.621652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:19.621153    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk" event={"ID":"7e10009d-973a-440c-aa8f-89aa1ce91755","Type":"ContainerStarted","Data":"e70e6642f8d0d10fb4abd070d2b2b464e292b4c9ad9b445ad268ef5c632ec5c2"}
Oct 30 13:13:19.621652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:19.621175    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"
Oct 30 13:13:19.673700 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:19.673639    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk" podStartSLOduration=1.673624624 podStartE2EDuration="1.673624624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:19.670220444 +0000 UTC m=+1641.253650156" watchObservedRunningTime="2026-10-30 13:13:19.673624624 +0000 UTC m=+1641.257054314"
Oct 30 13:13:22.027632 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:22.027597    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"]
Oct 30 13:13:22.079109 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:22.079072    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"]
Oct 30 13:13:27.028533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.028465    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5" podUID="d05d155d-5f05-4945-9d25-4be97a2420e1" containerName="payload-processing" containerID="cri-o://e94432ce22a9f10cafe9b51c0ca1f23d9771270ed352ab4e04de775863a08a23" gracePeriod=25
Oct 30 13:13:27.079556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.079488    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9" podUID="a08ff971-0181-4107-ad1c-61f9b0ebc70b" containerName="payload-pre-processing" containerID="cri-o://a4824a8359d4b13ce3bfe038056bf68646c6d1f5a31bc0e1d032d5df0e0d6cf9" gracePeriod=25
Oct 30 13:13:27.376397 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.376082    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"
Oct 30 13:13:27.393146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.393110    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d05d155d-5f05-4945-9d25-4be97a2420e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05d155d-5f05-4945-9d25-4be97a2420e1-plugins-config-volume\") pod \"d05d155d-5f05-4945-9d25-4be97a2420e1\" (UID: \"d05d155d-5f05-4945-9d25-4be97a2420e1\") "
Oct 30 13:13:27.393343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.393258    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d05d155d-5f05-4945-9d25-4be97a2420e1-kube-api-access-s5xhn\" (UniqueName: \"kubernetes.io/projected/d05d155d-5f05-4945-9d25-4be97a2420e1-kube-api-access-s5xhn\") pod \"d05d155d-5f05-4945-9d25-4be97a2420e1\" (UID: \"d05d155d-5f05-4945-9d25-4be97a2420e1\") "
Oct 30 13:13:27.393565 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.393541    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d05d155d-5f05-4945-9d25-4be97a2420e1-plugins-config-volume" pod "d05d155d-5f05-4945-9d25-4be97a2420e1" (UID: "d05d155d-5f05-4945-9d25-4be97a2420e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:13:27.394040 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.394018    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d05d155d-5f05-4945-9d25-4be97a2420e1-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:27.395632 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.395610    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d05d155d-5f05-4945-9d25-4be97a2420e1-kube-api-access-s5xhn" pod "d05d155d-5f05-4945-9d25-4be97a2420e1" (UID: "d05d155d-5f05-4945-9d25-4be97a2420e1"). InnerVolumeSpecName "kube-api-access-s5xhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:13:27.426580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.426552    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"
Oct 30 13:13:27.499299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.498657    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5xhn\" (UniqueName: \"kubernetes.io/projected/d05d155d-5f05-4945-9d25-4be97a2420e1-kube-api-access-s5xhn\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:27.602981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.599561    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08ff971-0181-4107-ad1c-61f9b0ebc70b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08ff971-0181-4107-ad1c-61f9b0ebc70b-plugins-config-volume\") pod \"a08ff971-0181-4107-ad1c-61f9b0ebc70b\" (UID: \"a08ff971-0181-4107-ad1c-61f9b0ebc70b\") "
Oct 30 13:13:27.602981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.599693    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08ff971-0181-4107-ad1c-61f9b0ebc70b-kube-api-access-bv64z\" (UniqueName: \"kubernetes.io/projected/a08ff971-0181-4107-ad1c-61f9b0ebc70b-kube-api-access-bv64z\") pod \"a08ff971-0181-4107-ad1c-61f9b0ebc70b\" (UID: \"a08ff971-0181-4107-ad1c-61f9b0ebc70b\") "
Oct 30 13:13:27.602981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.601690    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08ff971-0181-4107-ad1c-61f9b0ebc70b-plugins-config-volume" pod "a08ff971-0181-4107-ad1c-61f9b0ebc70b" (UID: "a08ff971-0181-4107-ad1c-61f9b0ebc70b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:13:27.608293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.605571    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08ff971-0181-4107-ad1c-61f9b0ebc70b-kube-api-access-bv64z" pod "a08ff971-0181-4107-ad1c-61f9b0ebc70b" (UID: "a08ff971-0181-4107-ad1c-61f9b0ebc70b"). InnerVolumeSpecName "kube-api-access-bv64z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:13:27.682589 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.682555    5893 generic.go:356] "Generic (PLEG): container finished" podID="d05d155d-5f05-4945-9d25-4be97a2420e1" containerID="e94432ce22a9f10cafe9b51c0ca1f23d9771270ed352ab4e04de775863a08a23" exitCode=0
Oct 30 13:13:27.682778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.682617    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"
Oct 30 13:13:27.682778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.682651    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5" event={"ID":"d05d155d-5f05-4945-9d25-4be97a2420e1","Type":"ContainerDied","Data":"e94432ce22a9f10cafe9b51c0ca1f23d9771270ed352ab4e04de775863a08a23"}
Oct 30 13:13:27.682778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.682717    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5" event={"ID":"d05d155d-5f05-4945-9d25-4be97a2420e1","Type":"ContainerDied","Data":"2f313ea03e9faba44a6dedb14acec22567bc43e34a1a6ca86df7c44206bb4321"}
Oct 30 13:13:27.682778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.682746    5893 scope.go:122] "RemoveContainer" containerID="e94432ce22a9f10cafe9b51c0ca1f23d9771270ed352ab4e04de775863a08a23"
Oct 30 13:13:27.685151 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.685022    5893 generic.go:356] "Generic (PLEG): container finished" podID="a08ff971-0181-4107-ad1c-61f9b0ebc70b" containerID="a4824a8359d4b13ce3bfe038056bf68646c6d1f5a31bc0e1d032d5df0e0d6cf9" exitCode=0
Oct 30 13:13:27.685151 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.685074    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"
Oct 30 13:13:27.685151 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.685091    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9" event={"ID":"a08ff971-0181-4107-ad1c-61f9b0ebc70b","Type":"ContainerDied","Data":"a4824a8359d4b13ce3bfe038056bf68646c6d1f5a31bc0e1d032d5df0e0d6cf9"}
Oct 30 13:13:27.685151 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.685138    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9" event={"ID":"a08ff971-0181-4107-ad1c-61f9b0ebc70b","Type":"ContainerDied","Data":"afd9f409d9182df724c064fea3024f066e8b52375a75b38c6c3ea1a8d3e9af8a"}
Oct 30 13:13:27.701449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.701422    5893 scope.go:122] "RemoveContainer" containerID="e94432ce22a9f10cafe9b51c0ca1f23d9771270ed352ab4e04de775863a08a23"
Oct 30 13:13:27.702115 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:13:27.702091    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e94432ce22a9f10cafe9b51c0ca1f23d9771270ed352ab4e04de775863a08a23\": container with ID starting with e94432ce22a9f10cafe9b51c0ca1f23d9771270ed352ab4e04de775863a08a23 not found: ID does not exist" containerID="e94432ce22a9f10cafe9b51c0ca1f23d9771270ed352ab4e04de775863a08a23"
Oct 30 13:13:27.702225 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.702127    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e94432ce22a9f10cafe9b51c0ca1f23d9771270ed352ab4e04de775863a08a23"} err="failed to get container status \"e94432ce22a9f10cafe9b51c0ca1f23d9771270ed352ab4e04de775863a08a23\": rpc error: code = NotFound desc = could not find container \"e94432ce22a9f10cafe9b51c0ca1f23d9771270ed352ab4e04de775863a08a23\": container with ID starting with e94432ce22a9f10cafe9b51c0ca1f23d9771270ed352ab4e04de775863a08a23 not found: ID does not exist"
Oct 30 13:13:27.702225 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.702151    5893 scope.go:122] "RemoveContainer" containerID="a4824a8359d4b13ce3bfe038056bf68646c6d1f5a31bc0e1d032d5df0e0d6cf9"
Oct 30 13:13:27.702518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.702495    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08ff971-0181-4107-ad1c-61f9b0ebc70b-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:27.702616 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.702520    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv64z\" (UniqueName: \"kubernetes.io/projected/a08ff971-0181-4107-ad1c-61f9b0ebc70b-kube-api-access-bv64z\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:27.711914 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.711898    5893 scope.go:122] "RemoveContainer" containerID="a4824a8359d4b13ce3bfe038056bf68646c6d1f5a31bc0e1d032d5df0e0d6cf9"
Oct 30 13:13:27.712189 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:13:27.712167    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4824a8359d4b13ce3bfe038056bf68646c6d1f5a31bc0e1d032d5df0e0d6cf9\": container with ID starting with a4824a8359d4b13ce3bfe038056bf68646c6d1f5a31bc0e1d032d5df0e0d6cf9 not found: ID does not exist" containerID="a4824a8359d4b13ce3bfe038056bf68646c6d1f5a31bc0e1d032d5df0e0d6cf9"
Oct 30 13:13:27.712318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.712194    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4824a8359d4b13ce3bfe038056bf68646c6d1f5a31bc0e1d032d5df0e0d6cf9"} err="failed to get container status \"a4824a8359d4b13ce3bfe038056bf68646c6d1f5a31bc0e1d032d5df0e0d6cf9\": rpc error: code = NotFound desc = could not find container \"a4824a8359d4b13ce3bfe038056bf68646c6d1f5a31bc0e1d032d5df0e0d6cf9\": container with ID starting with a4824a8359d4b13ce3bfe038056bf68646c6d1f5a31bc0e1d032d5df0e0d6cf9 not found: ID does not exist"
Oct 30 13:13:27.754659 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.754629    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"]
Oct 30 13:13:27.758082 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.758060    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1c0e6c42-dd9d66d84-8csv5"]
Oct 30 13:13:27.791378 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.791338    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"]
Oct 30 13:13:27.803353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:27.797222    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1c0e6c42-66d7d8479-hfgr9"]
Oct 30 13:13:28.542963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:28.542924    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a08ff971-0181-4107-ad1c-61f9b0ebc70b" path="/var/lib/kubelet/pods/a08ff971-0181-4107-ad1c-61f9b0ebc70b/volumes"
Oct 30 13:13:28.543657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:28.543635    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d05d155d-5f05-4945-9d25-4be97a2420e1" path="/var/lib/kubelet/pods/d05d155d-5f05-4945-9d25-4be97a2420e1/volumes"
Oct 30 13:13:30.632473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:30.632446    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"
Oct 30 13:13:42.534419 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:42.534378    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"]
Oct 30 13:13:42.534912 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:42.534634    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb" podUID="4b23c761-9c6f-487d-b7dd-4529d70cf6fe" containerName="istio-proxy" containerID="cri-o://ebb5a2c84e2488898ac8d9123cf4b24bd028ab9d9b9b20ce52aa15e5725a97cf" gracePeriod=30
Oct 30 13:13:47.855047 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.855016    5893 generic.go:356] "Generic (PLEG): container finished" podID="4b23c761-9c6f-487d-b7dd-4529d70cf6fe" containerID="ebb5a2c84e2488898ac8d9123cf4b24bd028ab9d9b9b20ce52aa15e5725a97cf" exitCode=0
Oct 30 13:13:47.855523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.855093    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb" event={"ID":"4b23c761-9c6f-487d-b7dd-4529d70cf6fe","Type":"ContainerDied","Data":"ebb5a2c84e2488898ac8d9123cf4b24bd028ab9d9b9b20ce52aa15e5725a97cf"}
Oct 30 13:13:47.883161 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.883130    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:13:47.997413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.997376    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-kube-api-access-ln9l4\" (UniqueName: \"kubernetes.io/projected/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-kube-api-access-ln9l4\") pod \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") "
Oct 30 13:13:47.997607 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.997494    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-token\" (UniqueName: \"kubernetes.io/projected/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-token\") pod \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") "
Oct 30 13:13:47.997607 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.997558    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istiod-ca-cert\") pod \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") "
Oct 30 13:13:47.997735 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.997619    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-envoy\") pod \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") "
Oct 30 13:13:47.997735 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.997683    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-podinfo\") pod \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") "
Oct 30 13:13:47.998051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.998016    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istiod-ca-cert" pod "4b23c761-9c6f-487d-b7dd-4529d70cf6fe" (UID: "4b23c761-9c6f-487d-b7dd-4529d70cf6fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:13:47.998174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.998134    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-workload-certs\") pod \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") "
Oct 30 13:13:47.998374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.998327    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-credential-socket\") pod \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") "
Oct 30 13:13:47.998519 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.998476    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-workload-socket\") pod \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") "
Oct 30 13:13:47.998578 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.998540    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-data\") pod \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\" (UID: \"4b23c761-9c6f-487d-b7dd-4529d70cf6fe\") "
Oct 30 13:13:47.999444 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.999419    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:47.999812 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:47.999784    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-data" pod "4b23c761-9c6f-487d-b7dd-4529d70cf6fe" (UID: "4b23c761-9c6f-487d-b7dd-4529d70cf6fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:13:48.000606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.000463    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-token" pod "4b23c761-9c6f-487d-b7dd-4529d70cf6fe" (UID: "4b23c761-9c6f-487d-b7dd-4529d70cf6fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:13:48.000606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.000624    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-kube-api-access-ln9l4" pod "4b23c761-9c6f-487d-b7dd-4529d70cf6fe" (UID: "4b23c761-9c6f-487d-b7dd-4529d70cf6fe"). InnerVolumeSpecName "kube-api-access-ln9l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:13:48.000606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.000662    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-credential-socket" pod "4b23c761-9c6f-487d-b7dd-4529d70cf6fe" (UID: "4b23c761-9c6f-487d-b7dd-4529d70cf6fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:13:48.000606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.000697    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-workload-socket" pod "4b23c761-9c6f-487d-b7dd-4529d70cf6fe" (UID: "4b23c761-9c6f-487d-b7dd-4529d70cf6fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:13:48.000606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.000888    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-envoy" pod "4b23c761-9c6f-487d-b7dd-4529d70cf6fe" (UID: "4b23c761-9c6f-487d-b7dd-4529d70cf6fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:13:48.000606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.000978    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-podinfo" pod "4b23c761-9c6f-487d-b7dd-4529d70cf6fe" (UID: "4b23c761-9c6f-487d-b7dd-4529d70cf6fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:13:48.000606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.001069    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-workload-certs" pod "4b23c761-9c6f-487d-b7dd-4529d70cf6fe" (UID: "4b23c761-9c6f-487d-b7dd-4529d70cf6fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:13:48.108541 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.105851    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:48.108541 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.105906    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln9l4\" (UniqueName: \"kubernetes.io/projected/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-kube-api-access-ln9l4\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:48.108541 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.105921    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:48.108541 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.105936    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:48.108541 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.105949    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:48.108541 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.105962    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:48.108541 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.105975    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:48.108541 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.105988    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b23c761-9c6f-487d-b7dd-4529d70cf6fe-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:48.660432 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:13:48.656722    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:13:48.866185 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.866101    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"
Oct 30 13:13:48.866185 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.866117    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb" event={"ID":"4b23c761-9c6f-487d-b7dd-4529d70cf6fe","Type":"ContainerDied","Data":"9604e3b5816c775ba0c3b71aa2207b0ff94c80eaa0e3cf60f0ee3dfdc5e57a74"}
Oct 30 13:13:48.866185 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.866185    5893 scope.go:122] "RemoveContainer" containerID="ebb5a2c84e2488898ac8d9123cf4b24bd028ab9d9b9b20ce52aa15e5725a97cf"
Oct 30 13:13:48.936053 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.936021    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"]
Oct 30 13:13:48.947566 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:48.947537    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-8xfqb"]
Oct 30 13:13:50.533738 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:50.533706    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b23c761-9c6f-487d-b7dd-4529d70cf6fe" path="/var/lib/kubelet/pods/4b23c761-9c6f-487d-b7dd-4529d70cf6fe/volumes"
Oct 30 13:13:50.652635 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:50.652593    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"]
Oct 30 13:13:50.652860 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:50.652833    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd" podUID="9fb9ed96-44c3-449d-90ea-055bd127799f" containerName="maas-api" containerID="cri-o://dc2197c9975e75b519a738f1454992c0eaac601b933a267e31cec294ad14f55a" gracePeriod=30
Oct 30 13:13:50.889553 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:50.889521    5893 generic.go:356] "Generic (PLEG): container finished" podID="9fb9ed96-44c3-449d-90ea-055bd127799f" containerID="dc2197c9975e75b519a738f1454992c0eaac601b933a267e31cec294ad14f55a" exitCode=0
Oct 30 13:13:50.889739 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:50.889685    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd" event={"ID":"9fb9ed96-44c3-449d-90ea-055bd127799f","Type":"ContainerDied","Data":"dc2197c9975e75b519a738f1454992c0eaac601b933a267e31cec294ad14f55a"}
Oct 30 13:13:51.021414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.021084    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:13:51.162358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.156179    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb9ed96-44c3-449d-90ea-055bd127799f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb9ed96-44c3-449d-90ea-055bd127799f-openshift-service-ca\") pod \"9fb9ed96-44c3-449d-90ea-055bd127799f\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") "
Oct 30 13:13:51.162358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.156357    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fb9ed96-44c3-449d-90ea-055bd127799f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fb9ed96-44c3-449d-90ea-055bd127799f-maas-api-tls\") pod \"9fb9ed96-44c3-449d-90ea-055bd127799f\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") "
Oct 30 13:13:51.162358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.156452    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fb9ed96-44c3-449d-90ea-055bd127799f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fb9ed96-44c3-449d-90ea-055bd127799f-metrics-certs\") pod \"9fb9ed96-44c3-449d-90ea-055bd127799f\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") "
Oct 30 13:13:51.162358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.156520    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb9ed96-44c3-449d-90ea-055bd127799f-kube-api-access-5jqgc\" (UniqueName: \"kubernetes.io/projected/9fb9ed96-44c3-449d-90ea-055bd127799f-kube-api-access-5jqgc\") pod \"9fb9ed96-44c3-449d-90ea-055bd127799f\" (UID: \"9fb9ed96-44c3-449d-90ea-055bd127799f\") "
Oct 30 13:13:51.186401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.169565    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb9ed96-44c3-449d-90ea-055bd127799f-kube-api-access-5jqgc" pod "9fb9ed96-44c3-449d-90ea-055bd127799f" (UID: "9fb9ed96-44c3-449d-90ea-055bd127799f"). InnerVolumeSpecName "kube-api-access-5jqgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:13:51.186401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.170130    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb9ed96-44c3-449d-90ea-055bd127799f-openshift-service-ca" pod "9fb9ed96-44c3-449d-90ea-055bd127799f" (UID: "9fb9ed96-44c3-449d-90ea-055bd127799f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:13:51.186401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.173194    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb9ed96-44c3-449d-90ea-055bd127799f-metrics-certs" pod "9fb9ed96-44c3-449d-90ea-055bd127799f" (UID: "9fb9ed96-44c3-449d-90ea-055bd127799f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:13:51.186401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.176928    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb9ed96-44c3-449d-90ea-055bd127799f-maas-api-tls" pod "9fb9ed96-44c3-449d-90ea-055bd127799f" (UID: "9fb9ed96-44c3-449d-90ea-055bd127799f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:13:51.261144 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.261104    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9fb9ed96-44c3-449d-90ea-055bd127799f-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:51.261144 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.261142    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9fb9ed96-44c3-449d-90ea-055bd127799f-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:51.261417 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.261157    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jqgc\" (UniqueName: \"kubernetes.io/projected/9fb9ed96-44c3-449d-90ea-055bd127799f-kube-api-access-5jqgc\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:51.261417 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.261172    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fb9ed96-44c3-449d-90ea-055bd127799f-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:13:51.930357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.930122    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"
Oct 30 13:13:51.930357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.930131    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd" event={"ID":"9fb9ed96-44c3-449d-90ea-055bd127799f","Type":"ContainerDied","Data":"557e584a6c18299abc27de8948c7abea17a3bf20b3b365600dbb17d1e9f8f1d2"}
Oct 30 13:13:51.930357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:51.930196    5893 scope.go:122] "RemoveContainer" containerID="dc2197c9975e75b519a738f1454992c0eaac601b933a267e31cec294ad14f55a"
Oct 30 13:13:52.008522 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.008322    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"]
Oct 30 13:13:52.017695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.017660    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1c0e6c42-6d7f6b784c-hdnsd"]
Oct 30 13:13:52.503980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.503947    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"]
Oct 30 13:13:52.505159 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.505136    5893 cpu_manager.go:414] "containerMap: removing container" podUID="9fb9ed96-44c3-449d-90ea-055bd127799f" containerName="maas-api"
Oct 30 13:13:52.505214 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.505164    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb9ed96-44c3-449d-90ea-055bd127799f" containerName="maas-api"
Oct 30 13:13:52.505214 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.505183    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d05d155d-5f05-4945-9d25-4be97a2420e1" containerName="payload-processing"
Oct 30 13:13:52.505214 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.505191    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d05d155d-5f05-4945-9d25-4be97a2420e1" containerName="payload-processing"
Oct 30 13:13:52.505351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.505225    5893 cpu_manager.go:414] "containerMap: removing container" podUID="a08ff971-0181-4107-ad1c-61f9b0ebc70b" containerName="payload-pre-processing"
Oct 30 13:13:52.505351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.505233    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08ff971-0181-4107-ad1c-61f9b0ebc70b" containerName="payload-pre-processing"
Oct 30 13:13:52.505351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.505255    5893 cpu_manager.go:414] "containerMap: removing container" podUID="4b23c761-9c6f-487d-b7dd-4529d70cf6fe" containerName="istio-proxy"
Oct 30 13:13:52.505351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.505262    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b23c761-9c6f-487d-b7dd-4529d70cf6fe" containerName="istio-proxy"
Oct 30 13:13:52.505487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.505473    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d05d155d-5f05-4945-9d25-4be97a2420e1" containerName="payload-processing"
Oct 30 13:13:52.505522 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.505494    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08ff971-0181-4107-ad1c-61f9b0ebc70b" containerName="payload-pre-processing"
Oct 30 13:13:52.505522 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.505508    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b23c761-9c6f-487d-b7dd-4529d70cf6fe" containerName="istio-proxy"
Oct 30 13:13:52.505581 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.505532    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb9ed96-44c3-449d-90ea-055bd127799f" containerName="maas-api"
Oct 30 13:13:52.509778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.509757    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"]
Oct 30 13:13:52.509936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.509909    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"
Oct 30 13:13:52.513160 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.513142    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"
Oct 30 13:13:52.535510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.535481    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8z96\" (UniqueName: \"kubernetes.io/projected/fca2cd17-7269-4207-be71-b2c5078a8e28-kube-api-access-q8z96\") pod \"payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm\" (UID: \"fca2cd17-7269-4207-be71-b2c5078a8e28\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"
Oct 30 13:13:52.535688 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.535673    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj66x\" (UniqueName: \"kubernetes.io/projected/d447e622-be3c-488d-b8ac-9a0713b0a643-kube-api-access-wj66x\") pod \"payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr\" (UID: \"d447e622-be3c-488d-b8ac-9a0713b0a643\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"
Oct 30 13:13:52.535891 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.535875    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca2cd17-7269-4207-be71-b2c5078a8e28-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm\" (UID: \"fca2cd17-7269-4207-be71-b2c5078a8e28\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"
Oct 30 13:13:52.535984 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.535970    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d447e622-be3c-488d-b8ac-9a0713b0a643-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr\" (UID: \"d447e622-be3c-488d-b8ac-9a0713b0a643\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"
Oct 30 13:13:52.536234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.536220    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4a8ee1-dockercfg-xr2xf\""
Oct 30 13:13:52.537636 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.537611    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb9ed96-44c3-449d-90ea-055bd127799f" path="/var/lib/kubelet/pods/9fb9ed96-44c3-449d-90ea-055bd127799f/volumes"
Oct 30 13:13:52.538340 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.538321    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4a8ee1\""
Oct 30 13:13:52.558908 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.555093    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"]
Oct 30 13:13:52.779230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.678127    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca2cd17-7269-4207-be71-b2c5078a8e28-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm\" (UID: \"fca2cd17-7269-4207-be71-b2c5078a8e28\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"
Oct 30 13:13:52.779230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.678504    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d447e622-be3c-488d-b8ac-9a0713b0a643-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr\" (UID: \"d447e622-be3c-488d-b8ac-9a0713b0a643\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"
Oct 30 13:13:52.779230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.678538    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8z96\" (UniqueName: \"kubernetes.io/projected/fca2cd17-7269-4207-be71-b2c5078a8e28-kube-api-access-q8z96\") pod \"payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm\" (UID: \"fca2cd17-7269-4207-be71-b2c5078a8e28\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"
Oct 30 13:13:52.779230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.678678    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj66x\" (UniqueName: \"kubernetes.io/projected/d447e622-be3c-488d-b8ac-9a0713b0a643-kube-api-access-wj66x\") pod \"payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr\" (UID: \"d447e622-be3c-488d-b8ac-9a0713b0a643\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"
Oct 30 13:13:52.779230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.768829    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca2cd17-7269-4207-be71-b2c5078a8e28-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm\" (UID: \"fca2cd17-7269-4207-be71-b2c5078a8e28\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"
Oct 30 13:13:52.779230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.771842    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d447e622-be3c-488d-b8ac-9a0713b0a643-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr\" (UID: \"d447e622-be3c-488d-b8ac-9a0713b0a643\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"
Oct 30 13:13:52.896225 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.896188    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"]
Oct 30 13:13:52.896437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.896400    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj66x\" (UniqueName: \"kubernetes.io/projected/d447e622-be3c-488d-b8ac-9a0713b0a643-kube-api-access-wj66x\") pod \"payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr\" (UID: \"d447e622-be3c-488d-b8ac-9a0713b0a643\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"
Oct 30 13:13:52.904043 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:52.904010    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8z96\" (UniqueName: \"kubernetes.io/projected/fca2cd17-7269-4207-be71-b2c5078a8e28-kube-api-access-q8z96\") pod \"payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm\" (UID: \"fca2cd17-7269-4207-be71-b2c5078a8e28\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"
Oct 30 13:13:53.143352 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:53.142566    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"
Oct 30 13:13:53.143352 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:53.143172    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"
Oct 30 13:13:53.566640 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:53.566559    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"]
Oct 30 13:13:53.574735 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:13:53.571466    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd447e622_be3c_488d_b8ac_9a0713b0a643.slice/crio-631f8b88ca781dbb2981212f632ba538af41118afe34db60a561dfe622e7bce7 WatchSource:0}: Error finding container 631f8b88ca781dbb2981212f632ba538af41118afe34db60a561dfe622e7bce7: Status 404 returned error can't find the container with id 631f8b88ca781dbb2981212f632ba538af41118afe34db60a561dfe622e7bce7
Oct 30 13:13:53.599799 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:53.599762    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"]
Oct 30 13:13:53.603153 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:13:53.603122    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca2cd17_7269_4207_be71_b2c5078a8e28.slice/crio-92b1db049588a6a9c5b9485f8cba74f696d40b690d70bb3ed92bb962441a4f8a WatchSource:0}: Error finding container 92b1db049588a6a9c5b9485f8cba74f696d40b690d70bb3ed92bb962441a4f8a: Status 404 returned error can't find the container with id 92b1db049588a6a9c5b9485f8cba74f696d40b690d70bb3ed92bb962441a4f8a
Oct 30 13:13:53.967595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:53.967554    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr" event={"ID":"d447e622-be3c-488d-b8ac-9a0713b0a643","Type":"ContainerStarted","Data":"a802146d310962a0e3e27a4a3226b279bbc7dc9ad0dbd643c1c362b7e10e0c78"}
Oct 30 13:13:53.967813 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:53.967622    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr" event={"ID":"d447e622-be3c-488d-b8ac-9a0713b0a643","Type":"ContainerStarted","Data":"631f8b88ca781dbb2981212f632ba538af41118afe34db60a561dfe622e7bce7"}
Oct 30 13:13:53.968406 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:53.968381    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"
Oct 30 13:13:53.970783 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:53.970716    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm" event={"ID":"fca2cd17-7269-4207-be71-b2c5078a8e28","Type":"ContainerStarted","Data":"74dc83cce5f91e1a97ed6139af7d9094171a6f3912cd4aedfc7779fca608920c"}
Oct 30 13:13:53.970783 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:53.970766    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm" event={"ID":"fca2cd17-7269-4207-be71-b2c5078a8e28","Type":"ContainerStarted","Data":"92b1db049588a6a9c5b9485f8cba74f696d40b690d70bb3ed92bb962441a4f8a"}
Oct 30 13:13:53.971546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:53.971524    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"
Oct 30 13:13:54.012085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:54.012027    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr" podStartSLOduration=2.012013418 podStartE2EDuration="2.012013418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:54.010177373 +0000 UTC m=+1675.593607065" watchObservedRunningTime="2026-10-30 13:13:54.012013418 +0000 UTC m=+1675.595443111"
Oct 30 13:13:54.059380 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:13:54.059264    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm" podStartSLOduration=2.059194293 podStartE2EDuration="2.059194293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:54.053689933 +0000 UTC m=+1675.637119626" watchObservedRunningTime="2026-10-30 13:13:54.059194293 +0000 UTC m=+1675.642623987"
Oct 30 13:14:04.982989 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:04.982961    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"
Oct 30 13:14:05.990891 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:05.990861    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"
Oct 30 13:14:08.335418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.335387    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"]
Oct 30 13:14:08.340406 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.340378    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:08.358316 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.358254    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-11cfa6-serving-cert\""
Oct 30 13:14:08.389686 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.389657    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"]
Oct 30 13:14:08.486394 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.486348    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16875e0-567d-4a21-916e-9d1338d15e14-maas-api-tls\") pod \"maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:08.486596 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.486551    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16875e0-567d-4a21-916e-9d1338d15e14-openshift-service-ca\") pod \"maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:08.486721 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.486707    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7gzj\" (UniqueName: \"kubernetes.io/projected/e16875e0-567d-4a21-916e-9d1338d15e14-kube-api-access-z7gzj\") pod \"maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:08.486788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.486754    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16875e0-567d-4a21-916e-9d1338d15e14-metrics-certs\") pod \"maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:08.592574 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.592462    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7gzj\" (UniqueName: \"kubernetes.io/projected/e16875e0-567d-4a21-916e-9d1338d15e14-kube-api-access-z7gzj\") pod \"maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:08.592574 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.592549    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16875e0-567d-4a21-916e-9d1338d15e14-metrics-certs\") pod \"maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:08.592797 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.592702    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16875e0-567d-4a21-916e-9d1338d15e14-maas-api-tls\") pod \"maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:08.592860 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.592831    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16875e0-567d-4a21-916e-9d1338d15e14-openshift-service-ca\") pod \"maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:08.594398 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.593732    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16875e0-567d-4a21-916e-9d1338d15e14-openshift-service-ca\") pod \"maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:08.598454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.598432    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16875e0-567d-4a21-916e-9d1338d15e14-maas-api-tls\") pod \"maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:08.601307 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.600348    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16875e0-567d-4a21-916e-9d1338d15e14-metrics-certs\") pod \"maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:08.618220 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.618186    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7gzj\" (UniqueName: \"kubernetes.io/projected/e16875e0-567d-4a21-916e-9d1338d15e14-kube-api-access-z7gzj\") pod \"maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:08.656764 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.656731    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:08.978201 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:08.978174    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"]
Oct 30 13:14:08.979921 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:14:08.979897    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16875e0_567d_4a21_916e_9d1338d15e14.slice/crio-4b5165b2226120fa722d0fa0bb75c481524dc413a7de081449fb7265135ce3e9 WatchSource:0}: Error finding container 4b5165b2226120fa722d0fa0bb75c481524dc413a7de081449fb7265135ce3e9: Status 404 returned error can't find the container with id 4b5165b2226120fa722d0fa0bb75c481524dc413a7de081449fb7265135ce3e9
Oct 30 13:14:09.117441 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:09.117409    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79" event={"ID":"e16875e0-567d-4a21-916e-9d1338d15e14","Type":"ContainerStarted","Data":"4b5165b2226120fa722d0fa0bb75c481524dc413a7de081449fb7265135ce3e9"}
Oct 30 13:14:10.127450 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:10.127414    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79" event={"ID":"e16875e0-567d-4a21-916e-9d1338d15e14","Type":"ContainerStarted","Data":"3a6e6263ed62b01e3ddbe1d171d9a1cc9b270b52fb0530d29d368f1f1b27ce20"}
Oct 30 13:14:10.127880 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:10.127513    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:10.218510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:10.218457    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79" podStartSLOduration=1.8896847270000001 podStartE2EDuration="2.218442524s" totalImagesPullingTime="328.757797ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:10.217980366 +0000 UTC m=+1691.801410059" watchObservedRunningTime="2026-10-30 13:14:10.218442524 +0000 UTC m=+1691.801872217"
Oct 30 13:14:16.166591 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:16.166547    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:14:18.698015 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:14:18.697977    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:14:25.175472 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.175425    5893 patch_prober.go:29] interesting pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:14:25.175985 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.175517    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng" podUID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:14:25.398301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.398258    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"]
Oct 30 13:14:25.401898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.401867    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.419588 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.419564    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-649303-openshift-default-dockercfg-vvzg7\""
Oct 30 13:14:25.441564 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.441483    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"]
Oct 30 13:14:25.528341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.520779    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-workload-socket\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.528341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.520830    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-token\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.528341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.520977    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-data\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.528341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.521019    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-podinfo\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.528341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.521190    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-credential-socket\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.528341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.521227    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhc9s\" (UniqueName: \"kubernetes.io/projected/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-kube-api-access-hhc9s\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.528341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.521254    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-workload-certs\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.528341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.521348    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-envoy\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.528341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.521422    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istiod-ca-cert\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.691384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.687585    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-workload-socket\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.691384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.687633    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-token\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.691384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.687773    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-data\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.691384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.687823    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-podinfo\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.691384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.687985    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-credential-socket\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.691384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.688019    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhc9s\" (UniqueName: \"kubernetes.io/projected/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-kube-api-access-hhc9s\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.691384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.688048    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-workload-certs\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.691384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.688121    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-envoy\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.691384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.688186    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istiod-ca-cert\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.691384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.689136    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istiod-ca-cert\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.691384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.689489    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-workload-socket\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.705120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.700826    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-data\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.705120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.701773    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-podinfo\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.707638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.707589    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-workload-certs\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.707638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.707601    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-credential-socket\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.717334 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.715560    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-envoy\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.722824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.722774    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-token\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:25.738821 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:25.738794    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhc9s\" (UniqueName: \"kubernetes.io/projected/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-kube-api-access-hhc9s\") pod \"e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") " pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:26.020506 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:26.020473    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:26.373903 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:26.373820    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"]
Oct 30 13:14:26.381096 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:14:26.381055    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebcad4e_6749_4f92_9f85_6e72eb1a1954.slice/crio-0d387daf50bc35979bf33ad646c6480ee105ee20a6bf60136a54d652ba145760 WatchSource:0}: Error finding container 0d387daf50bc35979bf33ad646c6480ee105ee20a6bf60136a54d652ba145760: Status 404 returned error can't find the container with id 0d387daf50bc35979bf33ad646c6480ee105ee20a6bf60136a54d652ba145760
Oct 30 13:14:26.385833 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:26.384481    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:14:26.385833 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:26.384581    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:14:26.385833 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:26.384631    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:14:26.449689 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:26.449660    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h" event={"ID":"1ebcad4e-6749-4f92-9f85-6e72eb1a1954","Type":"ContainerStarted","Data":"0d387daf50bc35979bf33ad646c6480ee105ee20a6bf60136a54d652ba145760"}
Oct 30 13:14:26.451677 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:26.451652    5893 generic.go:356] "Generic (PLEG): container finished" podID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerID="207ecee35167d3224270e421a47d925b1f95488a9ecd3ebc0cc9e037e32f2b68" exitCode=0
Oct 30 13:14:26.451827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:26.451721    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng" event={"ID":"d716f5ae-5dba-41f1-aeef-1d4fdeadba33","Type":"ContainerDied","Data":"207ecee35167d3224270e421a47d925b1f95488a9ecd3ebc0cc9e037e32f2b68"}
Oct 30 13:14:26.452206 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:26.452190    5893 scope.go:122] "RemoveContainer" containerID="207ecee35167d3224270e421a47d925b1f95488a9ecd3ebc0cc9e037e32f2b68"
Oct 30 13:14:26.454038 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:26.454009    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:14:26.454139 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:26.454078    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:14:26.454139 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:26.454110    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:14:27.486314 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:27.485815    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h" event={"ID":"1ebcad4e-6749-4f92-9f85-6e72eb1a1954","Type":"ContainerStarted","Data":"df7f17ae5f441d1ee4b66daffbcd1670bd2ce1084125d4ed86ef10bf1ab5d9e3"}
Oct 30 13:14:27.488881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:27.488829    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng" event={"ID":"d716f5ae-5dba-41f1-aeef-1d4fdeadba33","Type":"ContainerStarted","Data":"d570f8a6f067838b98627ad1032579dd49244382377eec0b3772ae06e341c54f"}
Oct 30 13:14:27.563298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:27.561636    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h" podStartSLOduration=2.561618936 podStartE2EDuration="2.561618936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:27.559213445 +0000 UTC m=+1709.142643150" watchObservedRunningTime="2026-10-30 13:14:27.561618936 +0000 UTC m=+1709.145048636"
Oct 30 13:14:28.021134 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:28.021102    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:28.027182 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:28.027156    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:28.364357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:28.358813    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:14:28.364357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:28.358983    5893 patch_prober.go:29] interesting pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:14:28.364357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:28.359027    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng" podUID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 13:14:28.498551 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:28.498522    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:28.499493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:28.499473    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:14:29.362301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:29.361364    5893 patch_prober.go:29] interesting pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:14:29.362301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:29.361423    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng" podUID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 13:14:30.360805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:30.360765    5893 patch_prober.go:29] interesting pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:14:30.361315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:30.360842    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng" podUID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 13:14:31.366298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:31.360637    5893 patch_prober.go:29] interesting pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:14:31.366298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:31.360709    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng" podUID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 13:14:32.385328 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:32.379428    5893 patch_prober.go:29] interesting pod/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body=
Oct 30 13:14:32.385328 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:32.379505    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng" podUID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused"
Oct 30 13:14:33.362555 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:33.362523    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:14:33.363045 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:33.363020    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:14:33.363757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:33.363731    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:14:42.000956 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:42.000873    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"]
Oct 30 13:14:42.001453 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:42.001085    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg" podUID="55a0fd69-b57d-48ba-af24-229454270b24" containerName="istio-proxy" containerID="cri-o://f2969764e1aa078a02dbe30119fac0d97c314a879791fe6ebb09a26e1bf35a9d" gracePeriod=30
Oct 30 13:14:46.247006 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:46.246975    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"]
Oct 30 13:14:46.247519 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:46.247178    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf" podUID="023728ea-7d4e-4d33-9618-320d46770bbd" containerName="istio-proxy" containerID="cri-o://58b27ecd451d82105d9740b06ebdd2f34e257f0f573cfba6cd8e0c31c95cb46d" gracePeriod=30
Oct 30 13:14:47.283397 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.283232    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:14:47.336618 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.336588    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-workload-socket\") pod \"55a0fd69-b57d-48ba-af24-229454270b24\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") "
Oct 30 13:14:47.336811 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.336653    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a0fd69-b57d-48ba-af24-229454270b24-istio-token\" (UniqueName: \"kubernetes.io/projected/55a0fd69-b57d-48ba-af24-229454270b24-istio-token\") pod \"55a0fd69-b57d-48ba-af24-229454270b24\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") "
Oct 30 13:14:47.336811 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.336691    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-credential-socket\") pod \"55a0fd69-b57d-48ba-af24-229454270b24\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") "
Oct 30 13:14:47.336811 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.336745    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-workload-certs\") pod \"55a0fd69-b57d-48ba-af24-229454270b24\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") "
Oct 30 13:14:47.336981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.336871    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55a0fd69-b57d-48ba-af24-229454270b24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a0fd69-b57d-48ba-af24-229454270b24-istiod-ca-cert\") pod \"55a0fd69-b57d-48ba-af24-229454270b24\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") "
Oct 30 13:14:47.336981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.336882    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-workload-socket" pod "55a0fd69-b57d-48ba-af24-229454270b24" (UID: "55a0fd69-b57d-48ba-af24-229454270b24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:47.336981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.336939    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55a0fd69-b57d-48ba-af24-229454270b24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a0fd69-b57d-48ba-af24-229454270b24-istio-podinfo\") pod \"55a0fd69-b57d-48ba-af24-229454270b24\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") "
Oct 30 13:14:47.336981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.336978    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55a0fd69-b57d-48ba-af24-229454270b24-kube-api-access-n5lm4\" (UniqueName: \"kubernetes.io/projected/55a0fd69-b57d-48ba-af24-229454270b24-kube-api-access-n5lm4\") pod \"55a0fd69-b57d-48ba-af24-229454270b24\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") "
Oct 30 13:14:47.337184 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.337069    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-istio-envoy\") pod \"55a0fd69-b57d-48ba-af24-229454270b24\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") "
Oct 30 13:14:47.337245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.337038    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-credential-socket" pod "55a0fd69-b57d-48ba-af24-229454270b24" (UID: "55a0fd69-b57d-48ba-af24-229454270b24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:47.337245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.337113    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-workload-certs" pod "55a0fd69-b57d-48ba-af24-229454270b24" (UID: "55a0fd69-b57d-48ba-af24-229454270b24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:47.337245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.337188    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-istio-data\") pod \"55a0fd69-b57d-48ba-af24-229454270b24\" (UID: \"55a0fd69-b57d-48ba-af24-229454270b24\") "
Oct 30 13:14:47.337245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.337229    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55a0fd69-b57d-48ba-af24-229454270b24-istiod-ca-cert" pod "55a0fd69-b57d-48ba-af24-229454270b24" (UID: "55a0fd69-b57d-48ba-af24-229454270b24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:47.337482 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.337434    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-istio-data" pod "55a0fd69-b57d-48ba-af24-229454270b24" (UID: "55a0fd69-b57d-48ba-af24-229454270b24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:47.337963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.337943    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:47.337963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.337963    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:47.338119 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.337972    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:47.338119 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.337983    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:47.338119 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.337991    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55a0fd69-b57d-48ba-af24-229454270b24-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:47.339556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.339530    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-istio-envoy" pod "55a0fd69-b57d-48ba-af24-229454270b24" (UID: "55a0fd69-b57d-48ba-af24-229454270b24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:47.339819 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.339786    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a0fd69-b57d-48ba-af24-229454270b24-kube-api-access-n5lm4" pod "55a0fd69-b57d-48ba-af24-229454270b24" (UID: "55a0fd69-b57d-48ba-af24-229454270b24"). InnerVolumeSpecName "kube-api-access-n5lm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:47.339819 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.339813    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55a0fd69-b57d-48ba-af24-229454270b24-istio-podinfo" pod "55a0fd69-b57d-48ba-af24-229454270b24" (UID: "55a0fd69-b57d-48ba-af24-229454270b24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:14:47.339954 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.339824    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55a0fd69-b57d-48ba-af24-229454270b24-istio-token" pod "55a0fd69-b57d-48ba-af24-229454270b24" (UID: "55a0fd69-b57d-48ba-af24-229454270b24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:47.439252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.439219    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55a0fd69-b57d-48ba-af24-229454270b24-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:47.439252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.439252    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55a0fd69-b57d-48ba-af24-229454270b24-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:47.439478 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.439298    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5lm4\" (UniqueName: \"kubernetes.io/projected/55a0fd69-b57d-48ba-af24-229454270b24-kube-api-access-n5lm4\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:47.439478 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.439311    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55a0fd69-b57d-48ba-af24-229454270b24-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:47.703467 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.703384    5893 generic.go:356] "Generic (PLEG): container finished" podID="55a0fd69-b57d-48ba-af24-229454270b24" containerID="f2969764e1aa078a02dbe30119fac0d97c314a879791fe6ebb09a26e1bf35a9d" exitCode=0
Oct 30 13:14:47.703467 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.703430    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg" event={"ID":"55a0fd69-b57d-48ba-af24-229454270b24","Type":"ContainerDied","Data":"f2969764e1aa078a02dbe30119fac0d97c314a879791fe6ebb09a26e1bf35a9d"}
Oct 30 13:14:47.703467 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.703461    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"
Oct 30 13:14:47.703722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.703483    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg" event={"ID":"55a0fd69-b57d-48ba-af24-229454270b24","Type":"ContainerDied","Data":"950dc1a5e4e5901d6905df4df96925c8c6b58c8e23db387d8d60332972bcb14f"}
Oct 30 13:14:47.703722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.703507    5893 scope.go:122] "RemoveContainer" containerID="f2969764e1aa078a02dbe30119fac0d97c314a879791fe6ebb09a26e1bf35a9d"
Oct 30 13:14:47.715563 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.715542    5893 scope.go:122] "RemoveContainer" containerID="f2969764e1aa078a02dbe30119fac0d97c314a879791fe6ebb09a26e1bf35a9d"
Oct 30 13:14:47.715920 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:14:47.715892    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2969764e1aa078a02dbe30119fac0d97c314a879791fe6ebb09a26e1bf35a9d\": container with ID starting with f2969764e1aa078a02dbe30119fac0d97c314a879791fe6ebb09a26e1bf35a9d not found: ID does not exist" containerID="f2969764e1aa078a02dbe30119fac0d97c314a879791fe6ebb09a26e1bf35a9d"
Oct 30 13:14:47.716025 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.715930    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2969764e1aa078a02dbe30119fac0d97c314a879791fe6ebb09a26e1bf35a9d"} err="failed to get container status \"f2969764e1aa078a02dbe30119fac0d97c314a879791fe6ebb09a26e1bf35a9d\": rpc error: code = NotFound desc = could not find container \"f2969764e1aa078a02dbe30119fac0d97c314a879791fe6ebb09a26e1bf35a9d\": container with ID starting with f2969764e1aa078a02dbe30119fac0d97c314a879791fe6ebb09a26e1bf35a9d not found: ID does not exist"
Oct 30 13:14:47.761374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.761317    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"]
Oct 30 13:14:47.767534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:47.767509    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-lh4zg"]
Oct 30 13:14:48.535032 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:48.535003    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55a0fd69-b57d-48ba-af24-229454270b24" path="/var/lib/kubelet/pods/55a0fd69-b57d-48ba-af24-229454270b24/volumes"
Oct 30 13:14:48.647996 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:14:48.647951    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:14:51.592010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.591979    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:14:51.706104 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.705541    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023728ea-7d4e-4d33-9618-320d46770bbd-kube-api-access-8n9cm\" (UniqueName: \"kubernetes.io/projected/023728ea-7d4e-4d33-9618-320d46770bbd-kube-api-access-8n9cm\") pod \"023728ea-7d4e-4d33-9618-320d46770bbd\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") "
Oct 30 13:14:51.706104 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.705594    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-workload-socket\") pod \"023728ea-7d4e-4d33-9618-320d46770bbd\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") "
Oct 30 13:14:51.706104 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.705641    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-istio-envoy\") pod \"023728ea-7d4e-4d33-9618-320d46770bbd\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") "
Oct 30 13:14:51.706104 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.705715    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-istio-data\") pod \"023728ea-7d4e-4d33-9618-320d46770bbd\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") "
Oct 30 13:14:51.706104 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.705832    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-workload-certs\") pod \"023728ea-7d4e-4d33-9618-320d46770bbd\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") "
Oct 30 13:14:51.706104 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.705892    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/023728ea-7d4e-4d33-9618-320d46770bbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023728ea-7d4e-4d33-9618-320d46770bbd-istiod-ca-cert\") pod \"023728ea-7d4e-4d33-9618-320d46770bbd\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") "
Oct 30 13:14:51.706104 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.705914    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-workload-socket" pod "023728ea-7d4e-4d33-9618-320d46770bbd" (UID: "023728ea-7d4e-4d33-9618-320d46770bbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:51.706104 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.706078    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-credential-socket\") pod \"023728ea-7d4e-4d33-9618-320d46770bbd\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") "
Oct 30 13:14:51.706104 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.706099    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-istio-data" pod "023728ea-7d4e-4d33-9618-320d46770bbd" (UID: "023728ea-7d4e-4d33-9618-320d46770bbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:51.707246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.706214    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/023728ea-7d4e-4d33-9618-320d46770bbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023728ea-7d4e-4d33-9618-320d46770bbd-istio-podinfo\") pod \"023728ea-7d4e-4d33-9618-320d46770bbd\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") "
Oct 30 13:14:51.707246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.706258    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023728ea-7d4e-4d33-9618-320d46770bbd-istio-token\" (UniqueName: \"kubernetes.io/projected/023728ea-7d4e-4d33-9618-320d46770bbd-istio-token\") pod \"023728ea-7d4e-4d33-9618-320d46770bbd\" (UID: \"023728ea-7d4e-4d33-9618-320d46770bbd\") "
Oct 30 13:14:51.707246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.706656    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-credential-socket" pod "023728ea-7d4e-4d33-9618-320d46770bbd" (UID: "023728ea-7d4e-4d33-9618-320d46770bbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:51.707246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.706722    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023728ea-7d4e-4d33-9618-320d46770bbd-istiod-ca-cert" pod "023728ea-7d4e-4d33-9618-320d46770bbd" (UID: "023728ea-7d4e-4d33-9618-320d46770bbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:51.707246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.706943    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-workload-certs" pod "023728ea-7d4e-4d33-9618-320d46770bbd" (UID: "023728ea-7d4e-4d33-9618-320d46770bbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:51.707542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.707398    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.707542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.707422    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.707542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.707437    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.707542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.707453    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.707542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.707467    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/023728ea-7d4e-4d33-9618-320d46770bbd-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.709129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.708859    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-istio-envoy" pod "023728ea-7d4e-4d33-9618-320d46770bbd" (UID: "023728ea-7d4e-4d33-9618-320d46770bbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:14:51.709741 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.709716    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023728ea-7d4e-4d33-9618-320d46770bbd-istio-token" pod "023728ea-7d4e-4d33-9618-320d46770bbd" (UID: "023728ea-7d4e-4d33-9618-320d46770bbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:51.709900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.709878    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/023728ea-7d4e-4d33-9618-320d46770bbd-istio-podinfo" pod "023728ea-7d4e-4d33-9618-320d46770bbd" (UID: "023728ea-7d4e-4d33-9618-320d46770bbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:14:51.710155 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.710135    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023728ea-7d4e-4d33-9618-320d46770bbd-kube-api-access-8n9cm" pod "023728ea-7d4e-4d33-9618-320d46770bbd" (UID: "023728ea-7d4e-4d33-9618-320d46770bbd"). InnerVolumeSpecName "kube-api-access-8n9cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:51.750099 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.750057    5893 generic.go:356] "Generic (PLEG): container finished" podID="023728ea-7d4e-4d33-9618-320d46770bbd" containerID="58b27ecd451d82105d9740b06ebdd2f34e257f0f573cfba6cd8e0c31c95cb46d" exitCode=0
Oct 30 13:14:51.750242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.750128    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf" event={"ID":"023728ea-7d4e-4d33-9618-320d46770bbd","Type":"ContainerDied","Data":"58b27ecd451d82105d9740b06ebdd2f34e257f0f573cfba6cd8e0c31c95cb46d"}
Oct 30 13:14:51.750242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.750162    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"
Oct 30 13:14:51.750242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.750183    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf" event={"ID":"023728ea-7d4e-4d33-9618-320d46770bbd","Type":"ContainerDied","Data":"7831d6e8cddad1676ab2ff44f104600fea9cb0880fb2b8b945fd3619fa0f1344"}
Oct 30 13:14:51.750242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.750212    5893 scope.go:122] "RemoveContainer" containerID="58b27ecd451d82105d9740b06ebdd2f34e257f0f573cfba6cd8e0c31c95cb46d"
Oct 30 13:14:51.766088 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.766066    5893 scope.go:122] "RemoveContainer" containerID="58b27ecd451d82105d9740b06ebdd2f34e257f0f573cfba6cd8e0c31c95cb46d"
Oct 30 13:14:51.766654 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:14:51.766623    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b27ecd451d82105d9740b06ebdd2f34e257f0f573cfba6cd8e0c31c95cb46d\": container with ID starting with 58b27ecd451d82105d9740b06ebdd2f34e257f0f573cfba6cd8e0c31c95cb46d not found: ID does not exist" containerID="58b27ecd451d82105d9740b06ebdd2f34e257f0f573cfba6cd8e0c31c95cb46d"
Oct 30 13:14:51.766779 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.766662    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b27ecd451d82105d9740b06ebdd2f34e257f0f573cfba6cd8e0c31c95cb46d"} err="failed to get container status \"58b27ecd451d82105d9740b06ebdd2f34e257f0f573cfba6cd8e0c31c95cb46d\": rpc error: code = NotFound desc = could not find container \"58b27ecd451d82105d9740b06ebdd2f34e257f0f573cfba6cd8e0c31c95cb46d\": container with ID starting with 58b27ecd451d82105d9740b06ebdd2f34e257f0f573cfba6cd8e0c31c95cb46d not found: ID does not exist"
Oct 30 13:14:51.808890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.808797    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/023728ea-7d4e-4d33-9618-320d46770bbd-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.808890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.808835    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/023728ea-7d4e-4d33-9618-320d46770bbd-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.808890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.808849    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n9cm\" (UniqueName: \"kubernetes.io/projected/023728ea-7d4e-4d33-9618-320d46770bbd-kube-api-access-8n9cm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.808890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.808862    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/023728ea-7d4e-4d33-9618-320d46770bbd-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.813640 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.813586    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"]
Oct 30 13:14:51.818492 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:51.818468    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1c0e6c42-openshift-default-6ffd6bbb65-br2xf"]
Oct 30 13:14:52.532581 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:52.532550    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="023728ea-7d4e-4d33-9618-320d46770bbd" path="/var/lib/kubelet/pods/023728ea-7d4e-4d33-9618-320d46770bbd/volumes"
Oct 30 13:14:58.019529 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:58.019494    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"]
Oct 30 13:14:58.020080 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:58.019729    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4" podUID="5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" containerName="istio-proxy" containerID="cri-o://1ba407d90a5e764a5a89a76e059e7fae2d5c65a2d9efc7ccb34e4920476cb5d2" gracePeriod=30
Oct 30 13:14:58.034909 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:58.034842    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"]
Oct 30 13:14:58.035106 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:14:58.035076    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw" podUID="0ff6f8a1-dc39-41ba-b620-60e3cd5723da" containerName="istio-proxy" containerID="cri-o://65d8ddcdd509cee474e85ce4ac37505173c1308eae7e3f9c67587a9cdbe6aa4a" gracePeriod=30
Oct 30 13:15:00.437121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.437041    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"]
Oct 30 13:15:00.441181 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.441147    5893 cpu_manager.go:414] "containerMap: removing container" podUID="023728ea-7d4e-4d33-9618-320d46770bbd" containerName="istio-proxy"
Oct 30 13:15:00.441360 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.441344    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023728ea-7d4e-4d33-9618-320d46770bbd" containerName="istio-proxy"
Oct 30 13:15:00.441487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.441475    5893 cpu_manager.go:414] "containerMap: removing container" podUID="55a0fd69-b57d-48ba-af24-229454270b24" containerName="istio-proxy"
Oct 30 13:15:00.441579 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.441567    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55a0fd69-b57d-48ba-af24-229454270b24" containerName="istio-proxy"
Oct 30 13:15:00.441885 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.441868    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="023728ea-7d4e-4d33-9618-320d46770bbd" containerName="istio-proxy"
Oct 30 13:15:00.441992 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.441980    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="55a0fd69-b57d-48ba-af24-229454270b24" containerName="istio-proxy"
Oct 30 13:15:00.448203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.448182    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"]
Oct 30 13:15:00.448355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.448331    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"
Oct 30 13:15:00.454486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.454452    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"
Oct 30 13:15:00.487709 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.487561    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl\" (UID: \"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"
Oct 30 13:15:00.487709 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.487663    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8grwj\" (UniqueName: \"kubernetes.io/projected/926867a1-95ec-4376-98ab-c570de7f46bc-kube-api-access-8grwj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt\" (UID: \"926867a1-95ec-4376-98ab-c570de7f46bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"
Oct 30 13:15:00.488495 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.487924    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf9wc\" (UniqueName: \"kubernetes.io/projected/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4-kube-api-access-qf9wc\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl\" (UID: \"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"
Oct 30 13:15:00.488495 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.487948    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926867a1-95ec-4376-98ab-c570de7f46bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt\" (UID: \"926867a1-95ec-4376-98ab-c570de7f46bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"
Oct 30 13:15:00.495884 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.495855    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"]
Oct 30 13:15:00.499875 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.499854    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:15:00.500372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.500208    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"]
Oct 30 13:15:00.514781 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.513820    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"]
Oct 30 13:15:00.527609 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.527584    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"
Oct 30 13:15:00.547984 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.547949    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"]
Oct 30 13:15:00.558406 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.558376    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"
Oct 30 13:15:00.568255 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.568225    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"]
Oct 30 13:15:00.578221 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.578197    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"]
Oct 30 13:15:00.589212 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.589173    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl\" (UID: \"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"
Oct 30 13:15:00.589362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.589223    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8grwj\" (UniqueName: \"kubernetes.io/projected/926867a1-95ec-4376-98ab-c570de7f46bc-kube-api-access-8grwj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt\" (UID: \"926867a1-95ec-4376-98ab-c570de7f46bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"
Oct 30 13:15:00.589362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.589256    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21aac6b7-5d00-4753-9159-9c8c31172f95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp\" (UID: \"21aac6b7-5d00-4753-9159-9c8c31172f95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"
Oct 30 13:15:00.589362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.589305    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d86n\" (UniqueName: \"kubernetes.io/projected/21aac6b7-5d00-4753-9159-9c8c31172f95-kube-api-access-5d86n\") pod \"maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp\" (UID: \"21aac6b7-5d00-4753-9159-9c8c31172f95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"
Oct 30 13:15:00.589626 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.589611    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8baef-b3fd-410f-a85c-60006c08c3ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm\" (UID: \"9dc8baef-b3fd-410f-a85c-60006c08c3ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"
Oct 30 13:15:00.589661 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.589645    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf9wc\" (UniqueName: \"kubernetes.io/projected/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4-kube-api-access-qf9wc\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl\" (UID: \"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"
Oct 30 13:15:00.589695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.589673    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926867a1-95ec-4376-98ab-c570de7f46bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt\" (UID: \"926867a1-95ec-4376-98ab-c570de7f46bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"
Oct 30 13:15:00.589761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.589735    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvpll\" (UniqueName: \"kubernetes.io/projected/9dc8baef-b3fd-410f-a85c-60006c08c3ad-kube-api-access-gvpll\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm\" (UID: \"9dc8baef-b3fd-410f-a85c-60006c08c3ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"
Oct 30 13:15:00.590757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.590735    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl\" (UID: \"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"
Oct 30 13:15:00.592512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.592491    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926867a1-95ec-4376-98ab-c570de7f46bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt\" (UID: \"926867a1-95ec-4376-98ab-c570de7f46bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"
Oct 30 13:15:00.635377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.635353    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8grwj\" (UniqueName: \"kubernetes.io/projected/926867a1-95ec-4376-98ab-c570de7f46bc-kube-api-access-8grwj\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt\" (UID: \"926867a1-95ec-4376-98ab-c570de7f46bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"
Oct 30 13:15:00.637234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.637207    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf9wc\" (UniqueName: \"kubernetes.io/projected/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4-kube-api-access-qf9wc\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl\" (UID: \"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"
Oct 30 13:15:00.691229 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.691148    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d86n\" (UniqueName: \"kubernetes.io/projected/21aac6b7-5d00-4753-9159-9c8c31172f95-kube-api-access-5d86n\") pod \"maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp\" (UID: \"21aac6b7-5d00-4753-9159-9c8c31172f95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"
Oct 30 13:15:00.691386 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.691350    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8baef-b3fd-410f-a85c-60006c08c3ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm\" (UID: \"9dc8baef-b3fd-410f-a85c-60006c08c3ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"
Oct 30 13:15:00.691446 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.691393    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvpll\" (UniqueName: \"kubernetes.io/projected/9dc8baef-b3fd-410f-a85c-60006c08c3ad-kube-api-access-gvpll\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm\" (UID: \"9dc8baef-b3fd-410f-a85c-60006c08c3ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"
Oct 30 13:15:00.691509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.691492    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21aac6b7-5d00-4753-9159-9c8c31172f95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp\" (UID: \"21aac6b7-5d00-4753-9159-9c8c31172f95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"
Oct 30 13:15:00.692004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.691976    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8baef-b3fd-410f-a85c-60006c08c3ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm\" (UID: \"9dc8baef-b3fd-410f-a85c-60006c08c3ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"
Oct 30 13:15:00.692114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.692044    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21aac6b7-5d00-4753-9159-9c8c31172f95-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp\" (UID: \"21aac6b7-5d00-4753-9159-9c8c31172f95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"
Oct 30 13:15:00.722413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.722385    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvpll\" (UniqueName: \"kubernetes.io/projected/9dc8baef-b3fd-410f-a85c-60006c08c3ad-kube-api-access-gvpll\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm\" (UID: \"9dc8baef-b3fd-410f-a85c-60006c08c3ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"
Oct 30 13:15:00.743187 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.743156    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d86n\" (UniqueName: \"kubernetes.io/projected/21aac6b7-5d00-4753-9159-9c8c31172f95-kube-api-access-5d86n\") pod \"maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp\" (UID: \"21aac6b7-5d00-4753-9159-9c8c31172f95\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"
Oct 30 13:15:00.775132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.775106    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"
Oct 30 13:15:00.784496 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.784472    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"
Oct 30 13:15:00.858435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.858380    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"
Oct 30 13:15:00.884333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:00.883811    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"
Oct 30 13:15:01.133602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:01.133577    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"]
Oct 30 13:15:01.135735 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:15:01.135709    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod926867a1_95ec_4376_98ab_c570de7f46bc.slice/crio-0725755f94299a2f41b55a6044bd8cde1af2ef6986d61860308d9a4f857b2025 WatchSource:0}: Error finding container 0725755f94299a2f41b55a6044bd8cde1af2ef6986d61860308d9a4f857b2025: Status 404 returned error can't find the container with id 0725755f94299a2f41b55a6044bd8cde1af2ef6986d61860308d9a4f857b2025
Oct 30 13:15:01.138456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:01.138415    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"]
Oct 30 13:15:01.243821 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:01.243778    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"]
Oct 30 13:15:01.244866 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:15:01.244840    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21aac6b7_5d00_4753_9159_9c8c31172f95.slice/crio-eacca07bccc5f9c49df1918568002ec2660ed42cd9d54ee089e61038870fa467 WatchSource:0}: Error finding container eacca07bccc5f9c49df1918568002ec2660ed42cd9d54ee089e61038870fa467: Status 404 returned error can't find the container with id eacca07bccc5f9c49df1918568002ec2660ed42cd9d54ee089e61038870fa467
Oct 30 13:15:01.266061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:01.266029    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"]
Oct 30 13:15:01.316165 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:15:01.316132    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dc8baef_b3fd_410f_a85c_60006c08c3ad.slice/crio-1b81fcbc8598a7c1d5ca42230661c5a093c5cd4c0a32df64b9eb5e8566d776f5 WatchSource:0}: Error finding container 1b81fcbc8598a7c1d5ca42230661c5a093c5cd4c0a32df64b9eb5e8566d776f5: Status 404 returned error can't find the container with id 1b81fcbc8598a7c1d5ca42230661c5a093c5cd4c0a32df64b9eb5e8566d776f5
Oct 30 13:15:01.868266 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:01.868226    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm" event={"ID":"9dc8baef-b3fd-410f-a85c-60006c08c3ad","Type":"ContainerStarted","Data":"1b81fcbc8598a7c1d5ca42230661c5a093c5cd4c0a32df64b9eb5e8566d776f5"}
Oct 30 13:15:01.871362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:01.871334    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt" event={"ID":"926867a1-95ec-4376-98ab-c570de7f46bc","Type":"ContainerStarted","Data":"0725755f94299a2f41b55a6044bd8cde1af2ef6986d61860308d9a4f857b2025"}
Oct 30 13:15:01.879649 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:01.879546    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp" event={"ID":"21aac6b7-5d00-4753-9159-9c8c31172f95","Type":"ContainerStarted","Data":"eacca07bccc5f9c49df1918568002ec2660ed42cd9d54ee089e61038870fa467"}
Oct 30 13:15:01.881721 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:01.881697    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl" event={"ID":"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4","Type":"ContainerStarted","Data":"09c59c1c028e9912ca66e6bdf165ca69386738346af1f09dd441fba065195c5c"}
Oct 30 13:15:03.468721 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.468694    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:15:03.487127 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.487091    5893 patch_prober.go:29] interesting pod/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 13:15:03.487322 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.487177    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw" podUID="0ff6f8a1-dc39-41ba-b620-60e3cd5723da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 13:15:03.542413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.542377    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-data\") pod \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") "
Oct 30 13:15:03.543023 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.542993    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-data" pod "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" (UID: "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:03.646695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.644910    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-podinfo\") pod \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") "
Oct 30 13:15:03.646695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.644971    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-envoy\") pod \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") "
Oct 30 13:15:03.646695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.645013    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-credential-socket\") pod \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") "
Oct 30 13:15:03.646695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.645052    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istiod-ca-cert\") pod \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") "
Oct 30 13:15:03.646695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.645070    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-token\" (UniqueName: \"kubernetes.io/projected/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-token\") pod \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") "
Oct 30 13:15:03.646695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.645123    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-kube-api-access-pbp4d\" (UniqueName: \"kubernetes.io/projected/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-kube-api-access-pbp4d\") pod \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") "
Oct 30 13:15:03.646695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.645264    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-workload-socket\") pod \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") "
Oct 30 13:15:03.646695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.645338    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-workload-certs\") pod \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\" (UID: \"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82\") "
Oct 30 13:15:03.646695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.645773    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.646695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.646296    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istiod-ca-cert" pod "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" (UID: "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:03.647374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.646843    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-workload-socket" pod "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" (UID: "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:03.647374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.646933    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-credential-socket" pod "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" (UID: "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:03.647374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.646972    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-workload-certs" pod "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" (UID: "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:03.682990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.682345    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-kube-api-access-pbp4d" pod "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" (UID: "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82"). InnerVolumeSpecName "kube-api-access-pbp4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:03.682990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.682811    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-podinfo" pod "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" (UID: "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:15:03.684946 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.684913    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-token" pod "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" (UID: "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:03.689881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.689821    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-envoy" pod "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" (UID: "5757084c-bd02-4b5e-95ed-7ffe5ef2bd82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:03.771538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.766060    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.771538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.766095    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.771538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.766108    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.771538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.766121    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbp4d\" (UniqueName: \"kubernetes.io/projected/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-kube-api-access-pbp4d\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.771538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.766135    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.771538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.766147    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.771538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.766160    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.771538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.766172    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.860761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.860728    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:15:03.866961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.866932    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-kube-api-access-pwphn\" (UniqueName: \"kubernetes.io/projected/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-kube-api-access-pwphn\") pod \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") "
Oct 30 13:15:03.867127 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.867026    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-credential-socket\") pod \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") "
Oct 30 13:15:03.867207 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.867121    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-workload-certs\") pod \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") "
Oct 30 13:15:03.867207 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.867176    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-data\") pod \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") "
Oct 30 13:15:03.867392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.867219    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-podinfo\") pod \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") "
Oct 30 13:15:03.867392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.867246    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istiod-ca-cert\") pod \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") "
Oct 30 13:15:03.867392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.867361    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-token\" (UniqueName: \"kubernetes.io/projected/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-token\") pod \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") "
Oct 30 13:15:03.867552 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.867447    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-envoy\") pod \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") "
Oct 30 13:15:03.867552 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.867500    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-workload-socket\") pod \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\" (UID: \"0ff6f8a1-dc39-41ba-b620-60e3cd5723da\") "
Oct 30 13:15:03.867660 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.867576    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-workload-certs" pod "0ff6f8a1-dc39-41ba-b620-60e3cd5723da" (UID: "0ff6f8a1-dc39-41ba-b620-60e3cd5723da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:03.867910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.867862    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-credential-socket" pod "0ff6f8a1-dc39-41ba-b620-60e3cd5723da" (UID: "0ff6f8a1-dc39-41ba-b620-60e3cd5723da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:03.868816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.868687    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-data" pod "0ff6f8a1-dc39-41ba-b620-60e3cd5723da" (UID: "0ff6f8a1-dc39-41ba-b620-60e3cd5723da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:03.868816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.868759    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-workload-socket" pod "0ff6f8a1-dc39-41ba-b620-60e3cd5723da" (UID: "0ff6f8a1-dc39-41ba-b620-60e3cd5723da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:03.868997 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.868934    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istiod-ca-cert" pod "0ff6f8a1-dc39-41ba-b620-60e3cd5723da" (UID: "0ff6f8a1-dc39-41ba-b620-60e3cd5723da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:03.869805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.869711    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.869805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.869745    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.869805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.869760    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.869805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.869773    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.869805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.869786    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.870311 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.870286    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-kube-api-access-pwphn" pod "0ff6f8a1-dc39-41ba-b620-60e3cd5723da" (UID: "0ff6f8a1-dc39-41ba-b620-60e3cd5723da"). InnerVolumeSpecName "kube-api-access-pwphn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:03.883322 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.880683    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-token" pod "0ff6f8a1-dc39-41ba-b620-60e3cd5723da" (UID: "0ff6f8a1-dc39-41ba-b620-60e3cd5723da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:03.883322 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.880883    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-podinfo" pod "0ff6f8a1-dc39-41ba-b620-60e3cd5723da" (UID: "0ff6f8a1-dc39-41ba-b620-60e3cd5723da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:15:03.883322 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.882863    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-envoy" pod "0ff6f8a1-dc39-41ba-b620-60e3cd5723da" (UID: "0ff6f8a1-dc39-41ba-b620-60e3cd5723da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:03.907373 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.907335    5893 generic.go:356] "Generic (PLEG): container finished" podID="5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" containerID="1ba407d90a5e764a5a89a76e059e7fae2d5c65a2d9efc7ccb34e4920476cb5d2" exitCode=0
Oct 30 13:15:03.907554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.907517    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4" event={"ID":"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82","Type":"ContainerDied","Data":"1ba407d90a5e764a5a89a76e059e7fae2d5c65a2d9efc7ccb34e4920476cb5d2"}
Oct 30 13:15:03.907623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.907569    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4" event={"ID":"5757084c-bd02-4b5e-95ed-7ffe5ef2bd82","Type":"ContainerDied","Data":"b004047c0a55fcd2ce1b2cc9038ed02a91f02498df195602d58dd1fa284d79eb"}
Oct 30 13:15:03.907623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.907602    5893 scope.go:122] "RemoveContainer" containerID="1ba407d90a5e764a5a89a76e059e7fae2d5c65a2d9efc7ccb34e4920476cb5d2"
Oct 30 13:15:03.907791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.907774    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"
Oct 30 13:15:03.916611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.913188    5893 generic.go:356] "Generic (PLEG): container finished" podID="0ff6f8a1-dc39-41ba-b620-60e3cd5723da" containerID="65d8ddcdd509cee474e85ce4ac37505173c1308eae7e3f9c67587a9cdbe6aa4a" exitCode=0
Oct 30 13:15:03.916611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.913245    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw" event={"ID":"0ff6f8a1-dc39-41ba-b620-60e3cd5723da","Type":"ContainerDied","Data":"65d8ddcdd509cee474e85ce4ac37505173c1308eae7e3f9c67587a9cdbe6aa4a"}
Oct 30 13:15:03.916611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.913320    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw" event={"ID":"0ff6f8a1-dc39-41ba-b620-60e3cd5723da","Type":"ContainerDied","Data":"569b161f3eea8e88c59a2571725802573b0012380d4b246cada37980c50a6363"}
Oct 30 13:15:03.916611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.913394    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"
Oct 30 13:15:03.955687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.940606    5893 scope.go:122] "RemoveContainer" containerID="1ba407d90a5e764a5a89a76e059e7fae2d5c65a2d9efc7ccb34e4920476cb5d2"
Oct 30 13:15:03.955687 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:03.941072    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ba407d90a5e764a5a89a76e059e7fae2d5c65a2d9efc7ccb34e4920476cb5d2\": container with ID starting with 1ba407d90a5e764a5a89a76e059e7fae2d5c65a2d9efc7ccb34e4920476cb5d2 not found: ID does not exist" containerID="1ba407d90a5e764a5a89a76e059e7fae2d5c65a2d9efc7ccb34e4920476cb5d2"
Oct 30 13:15:03.955687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.941116    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ba407d90a5e764a5a89a76e059e7fae2d5c65a2d9efc7ccb34e4920476cb5d2"} err="failed to get container status \"1ba407d90a5e764a5a89a76e059e7fae2d5c65a2d9efc7ccb34e4920476cb5d2\": rpc error: code = NotFound desc = could not find container \"1ba407d90a5e764a5a89a76e059e7fae2d5c65a2d9efc7ccb34e4920476cb5d2\": container with ID starting with 1ba407d90a5e764a5a89a76e059e7fae2d5c65a2d9efc7ccb34e4920476cb5d2 not found: ID does not exist"
Oct 30 13:15:03.955687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.941140    5893 scope.go:122] "RemoveContainer" containerID="65d8ddcdd509cee474e85ce4ac37505173c1308eae7e3f9c67587a9cdbe6aa4a"
Oct 30 13:15:03.971953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.971817    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwphn\" (UniqueName: \"kubernetes.io/projected/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-kube-api-access-pwphn\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.971953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.971844    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.971953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.971859    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.971953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.971872    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff6f8a1-dc39-41ba-b620-60e3cd5723da-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:03.972191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.972058    5893 scope.go:122] "RemoveContainer" containerID="65d8ddcdd509cee474e85ce4ac37505173c1308eae7e3f9c67587a9cdbe6aa4a"
Oct 30 13:15:03.979303 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:03.979257    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d8ddcdd509cee474e85ce4ac37505173c1308eae7e3f9c67587a9cdbe6aa4a\": container with ID starting with 65d8ddcdd509cee474e85ce4ac37505173c1308eae7e3f9c67587a9cdbe6aa4a not found: ID does not exist" containerID="65d8ddcdd509cee474e85ce4ac37505173c1308eae7e3f9c67587a9cdbe6aa4a"
Oct 30 13:15:03.979526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:03.979492    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d8ddcdd509cee474e85ce4ac37505173c1308eae7e3f9c67587a9cdbe6aa4a"} err="failed to get container status \"65d8ddcdd509cee474e85ce4ac37505173c1308eae7e3f9c67587a9cdbe6aa4a\": rpc error: code = NotFound desc = could not find container \"65d8ddcdd509cee474e85ce4ac37505173c1308eae7e3f9c67587a9cdbe6aa4a\": container with ID starting with 65d8ddcdd509cee474e85ce4ac37505173c1308eae7e3f9c67587a9cdbe6aa4a not found: ID does not exist"
Oct 30 13:15:04.055113 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:04.055048    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"]
Oct 30 13:15:04.062010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:04.061935    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-n6bvw"]
Oct 30 13:15:04.192877 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:04.160258    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"]
Oct 30 13:15:04.203424 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:04.203395    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-22pb4"]
Oct 30 13:15:04.538094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:04.538036    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff6f8a1-dc39-41ba-b620-60e3cd5723da" path="/var/lib/kubelet/pods/0ff6f8a1-dc39-41ba-b620-60e3cd5723da/volumes"
Oct 30 13:15:04.538955 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:04.538929    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" path="/var/lib/kubelet/pods/5757084c-bd02-4b5e-95ed-7ffe5ef2bd82/volumes"
Oct 30 13:15:05.904301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:05.903001    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"]
Oct 30 13:15:05.904301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:05.904110    5893 cpu_manager.go:414] "containerMap: removing container" podUID="0ff6f8a1-dc39-41ba-b620-60e3cd5723da" containerName="istio-proxy"
Oct 30 13:15:05.904301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:05.904130    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff6f8a1-dc39-41ba-b620-60e3cd5723da" containerName="istio-proxy"
Oct 30 13:15:05.904301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:05.904212    5893 cpu_manager.go:414] "containerMap: removing container" podUID="5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" containerName="istio-proxy"
Oct 30 13:15:05.904301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:05.904220    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" containerName="istio-proxy"
Oct 30 13:15:05.905087 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:05.904423    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="5757084c-bd02-4b5e-95ed-7ffe5ef2bd82" containerName="istio-proxy"
Oct 30 13:15:05.905087 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:05.904449    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff6f8a1-dc39-41ba-b620-60e3cd5723da" containerName="istio-proxy"
Oct 30 13:15:05.955298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:05.955141    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"]
Oct 30 13:15:05.955464 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:05.955310    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"
Oct 30 13:15:05.992237 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:05.992208    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:15:06.086249 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.086212    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783977cb-aa63-461f-9fb0-ee490af3989c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr\" (UID: \"783977cb-aa63-461f-9fb0-ee490af3989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"
Oct 30 13:15:06.086573 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.086547    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783977cb-aa63-461f-9fb0-ee490af3989c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr\" (UID: \"783977cb-aa63-461f-9fb0-ee490af3989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"
Oct 30 13:15:06.086975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.086950    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxxwx\" (UniqueName: \"kubernetes.io/projected/783977cb-aa63-461f-9fb0-ee490af3989c-kube-api-access-xxxwx\") pod \"maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr\" (UID: \"783977cb-aa63-461f-9fb0-ee490af3989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"
Oct 30 13:15:06.190343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.189307    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxxwx\" (UniqueName: \"kubernetes.io/projected/783977cb-aa63-461f-9fb0-ee490af3989c-kube-api-access-xxxwx\") pod \"maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr\" (UID: \"783977cb-aa63-461f-9fb0-ee490af3989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"
Oct 30 13:15:06.190343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.189440    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783977cb-aa63-461f-9fb0-ee490af3989c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr\" (UID: \"783977cb-aa63-461f-9fb0-ee490af3989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"
Oct 30 13:15:06.190343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.189529    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783977cb-aa63-461f-9fb0-ee490af3989c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr\" (UID: \"783977cb-aa63-461f-9fb0-ee490af3989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"
Oct 30 13:15:06.195226 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.195166    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783977cb-aa63-461f-9fb0-ee490af3989c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr\" (UID: \"783977cb-aa63-461f-9fb0-ee490af3989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"
Oct 30 13:15:06.200649 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.200612    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783977cb-aa63-461f-9fb0-ee490af3989c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr\" (UID: \"783977cb-aa63-461f-9fb0-ee490af3989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"
Oct 30 13:15:06.235564 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.235527    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxxwx\" (UniqueName: \"kubernetes.io/projected/783977cb-aa63-461f-9fb0-ee490af3989c-kube-api-access-xxxwx\") pod \"maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr\" (UID: \"783977cb-aa63-461f-9fb0-ee490af3989c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"
Oct 30 13:15:06.286477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.286267    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"
Oct 30 13:15:06.420959 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.417237    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"]
Oct 30 13:15:06.425411 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.425386    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"
Oct 30 13:15:06.429303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.428879    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"]
Oct 30 13:15:06.525299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.510076    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6af611-dockercfg-8s56n\""
Oct 30 13:15:06.525299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.510889    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6af611\""
Oct 30 13:15:06.642541 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.639860    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"
Oct 30 13:15:06.647384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.644957    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwxsh\" (UniqueName: \"kubernetes.io/projected/42c820c0-bd02-40ac-b278-8974ade564b4-kube-api-access-rwxsh\") pod \"payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r\" (UID: \"42c820c0-bd02-40ac-b278-8974ade564b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"
Oct 30 13:15:06.647384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.645088    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c820c0-bd02-40ac-b278-8974ade564b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r\" (UID: \"42c820c0-bd02-40ac-b278-8974ade564b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"
Oct 30 13:15:06.873878 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.871841    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce126ff4-a811-4d1e-83d0-3037add33d2e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m\" (UID: \"ce126ff4-a811-4d1e-83d0-3037add33d2e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"
Oct 30 13:15:06.873878 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.872097    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwxsh\" (UniqueName: \"kubernetes.io/projected/42c820c0-bd02-40ac-b278-8974ade564b4-kube-api-access-rwxsh\") pod \"payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r\" (UID: \"42c820c0-bd02-40ac-b278-8974ade564b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"
Oct 30 13:15:06.873878 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.872236    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c820c0-bd02-40ac-b278-8974ade564b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r\" (UID: \"42c820c0-bd02-40ac-b278-8974ade564b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"
Oct 30 13:15:06.873878 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.872314    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2h78\" (UniqueName: \"kubernetes.io/projected/ce126ff4-a811-4d1e-83d0-3037add33d2e-kube-api-access-d2h78\") pod \"payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m\" (UID: \"ce126ff4-a811-4d1e-83d0-3037add33d2e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"
Oct 30 13:15:06.873878 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.873832    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c820c0-bd02-40ac-b278-8974ade564b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r\" (UID: \"42c820c0-bd02-40ac-b278-8974ade564b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"
Oct 30 13:15:06.925701 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:06.925656    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwxsh\" (UniqueName: \"kubernetes.io/projected/42c820c0-bd02-40ac-b278-8974ade564b4-kube-api-access-rwxsh\") pod \"payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r\" (UID: \"42c820c0-bd02-40ac-b278-8974ade564b4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"
Oct 30 13:15:07.004088 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:07.003858    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"
Oct 30 13:15:07.036828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:07.036786    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"]
Oct 30 13:15:07.037027 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:07.036840    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"]
Oct 30 13:15:07.071828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:07.038469    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2h78\" (UniqueName: \"kubernetes.io/projected/ce126ff4-a811-4d1e-83d0-3037add33d2e-kube-api-access-d2h78\") pod \"payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m\" (UID: \"ce126ff4-a811-4d1e-83d0-3037add33d2e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"
Oct 30 13:15:07.072082 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:07.072046    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce126ff4-a811-4d1e-83d0-3037add33d2e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m\" (UID: \"ce126ff4-a811-4d1e-83d0-3037add33d2e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"
Oct 30 13:15:07.073780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:07.073738    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce126ff4-a811-4d1e-83d0-3037add33d2e-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m\" (UID: \"ce126ff4-a811-4d1e-83d0-3037add33d2e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"
Oct 30 13:15:07.073906 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:07.066594    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr" event={"ID":"783977cb-aa63-461f-9fb0-ee490af3989c","Type":"ContainerStarted","Data":"0b1b290598827f70970a5aefeebea5669881fbbf208b4335599d107018268bb2"}
Oct 30 13:15:07.078479 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:07.078453    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"]
Oct 30 13:15:07.101905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:07.101869    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2h78\" (UniqueName: \"kubernetes.io/projected/ce126ff4-a811-4d1e-83d0-3037add33d2e-kube-api-access-d2h78\") pod \"payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m\" (UID: \"ce126ff4-a811-4d1e-83d0-3037add33d2e\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"
Oct 30 13:15:07.322457 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:07.322424    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"
Oct 30 13:15:08.584427 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:08.565199    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:15:08.726372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:08.721126    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:15:08.907350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:08.906973    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd" podUID="e2b6f77b-f0bc-4e73-9572-2e652b9f58ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:15:09.015299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:09.004227    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"]
Oct 30 13:15:09.020443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:09.019742    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:15:09.234129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:09.234068    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"]
Oct 30 13:15:09.239439 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:15:09.239409    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c820c0_bd02_40ac_b278_8974ade564b4.slice/crio-7ef6847632ab013242ccf3c4b19d5a86efe8e0f6000e0b58d36bfdfe64a2dcc7 WatchSource:0}: Error finding container 7ef6847632ab013242ccf3c4b19d5a86efe8e0f6000e0b58d36bfdfe64a2dcc7: Status 404 returned error can't find the container with id 7ef6847632ab013242ccf3c4b19d5a86efe8e0f6000e0b58d36bfdfe64a2dcc7
Oct 30 13:15:09.454466 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:09.454361    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r" event={"ID":"42c820c0-bd02-40ac-b278-8974ade564b4","Type":"ContainerStarted","Data":"7ef6847632ab013242ccf3c4b19d5a86efe8e0f6000e0b58d36bfdfe64a2dcc7"}
Oct 30 13:15:09.457191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:09.457155    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m" event={"ID":"ce126ff4-a811-4d1e-83d0-3037add33d2e","Type":"ContainerStarted","Data":"55c11d7d7dd67d6cb2f6f819a2baa723704873d53143576eb467cd79aa309cfd"}
Oct 30 13:15:09.457366 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:09.457219    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m" event={"ID":"ce126ff4-a811-4d1e-83d0-3037add33d2e","Type":"ContainerStarted","Data":"b137cd8c5210f68e2640c79656d4f3924cc2e611caa4280a09b0ba49ee320b65"}
Oct 30 13:15:09.457366 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:09.457285    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"
Oct 30 13:15:09.505326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:09.505252    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m" podStartSLOduration=3.505233421 podStartE2EDuration="3.505233421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:09.502875228 +0000 UTC m=+1751.086304938" watchObservedRunningTime="2026-10-30 13:15:09.505233421 +0000 UTC m=+1751.088663115"
Oct 30 13:15:10.468570 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:10.468495    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r" event={"ID":"42c820c0-bd02-40ac-b278-8974ade564b4","Type":"ContainerStarted","Data":"87689cb355dd3766836d309aa39144f503e744af4fc056705f1ada6758d9a78a"}
Oct 30 13:15:10.468988 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:10.468588    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"
Oct 30 13:15:10.470608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:10.470563    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt" event={"ID":"926867a1-95ec-4376-98ab-c570de7f46bc","Type":"ContainerStarted","Data":"9563a36321a0e0c7cef8ccd47a64c7a4dbc23bf2645720d9203d476397cd5336"}
Oct 30 13:15:10.526999 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:10.526913    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r" podStartSLOduration=4.5268997429999995 podStartE2EDuration="4.526899743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:10.524881931 +0000 UTC m=+1752.108311626" watchObservedRunningTime="2026-10-30 13:15:10.526899743 +0000 UTC m=+1752.110329437"
Oct 30 13:15:10.579017 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:10.578969    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt" podStartSLOduration=1.9471733740000001 podStartE2EDuration="10.57895655s" totalImagesPullingTime="8.631783176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:10.575709946 +0000 UTC m=+1752.159139640" watchObservedRunningTime="2026-10-30 13:15:10.57895655 +0000 UTC m=+1752.162386244"
Oct 30 13:15:11.480838 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:11.480800    5893 generic.go:356] "Generic (PLEG): container finished" podID="926867a1-95ec-4376-98ab-c570de7f46bc" containerID="9563a36321a0e0c7cef8ccd47a64c7a4dbc23bf2645720d9203d476397cd5336" exitCode=60
Oct 30 13:15:11.481349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:11.480890    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt" event={"ID":"926867a1-95ec-4376-98ab-c570de7f46bc","Type":"ContainerDied","Data":"9563a36321a0e0c7cef8ccd47a64c7a4dbc23bf2645720d9203d476397cd5336"}
Oct 30 13:15:11.481349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:11.481262    5893 scope.go:122] "RemoveContainer" containerID="9563a36321a0e0c7cef8ccd47a64c7a4dbc23bf2645720d9203d476397cd5336"
Oct 30 13:15:11.482993 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:11.482968    5893 generic.go:356] "Generic (PLEG): container finished" podID="21aac6b7-5d00-4753-9159-9c8c31172f95" containerID="cdaadf4d599a929e5558c18d9ce55aedadd4439e6a593faa4aefd99becdec134" exitCode=60
Oct 30 13:15:11.483206 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:11.483007    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp" event={"ID":"21aac6b7-5d00-4753-9159-9c8c31172f95","Type":"ContainerDied","Data":"cdaadf4d599a929e5558c18d9ce55aedadd4439e6a593faa4aefd99becdec134"}
Oct 30 13:15:11.483394 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:11.483366    5893 scope.go:122] "RemoveContainer" containerID="cdaadf4d599a929e5558c18d9ce55aedadd4439e6a593faa4aefd99becdec134"
Oct 30 13:15:11.486783 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:11.486307    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr" event={"ID":"783977cb-aa63-461f-9fb0-ee490af3989c","Type":"ContainerStarted","Data":"c57a16a9dba0f43d11f1ba6c0f684cdd48e045e44ddd391a40be6bb24653d2fd"}
Oct 30 13:15:11.488603 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:11.488513    5893 generic.go:356] "Generic (PLEG): container finished" podID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" containerID="9e9c9615b5800d0677427becddf39a04a8e4b9917ffbe849b25a473f4ac9fe10" exitCode=60
Oct 30 13:15:11.488651 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:11.488593    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl" event={"ID":"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4","Type":"ContainerDied","Data":"9e9c9615b5800d0677427becddf39a04a8e4b9917ffbe849b25a473f4ac9fe10"}
Oct 30 13:15:11.488839 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:11.488813    5893 scope.go:122] "RemoveContainer" containerID="9e9c9615b5800d0677427becddf39a04a8e4b9917ffbe849b25a473f4ac9fe10"
Oct 30 13:15:11.490414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:11.490393    5893 generic.go:356] "Generic (PLEG): container finished" podID="9dc8baef-b3fd-410f-a85c-60006c08c3ad" containerID="4c74c251269b88e8ff23e8cd8c4d04477e7d3c79cb47eb6e125943f147a190d0" exitCode=60
Oct 30 13:15:11.490501 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:11.490468    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm" event={"ID":"9dc8baef-b3fd-410f-a85c-60006c08c3ad","Type":"ContainerDied","Data":"4c74c251269b88e8ff23e8cd8c4d04477e7d3c79cb47eb6e125943f147a190d0"}
Oct 30 13:15:11.490740 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:11.490725    5893 scope.go:122] "RemoveContainer" containerID="4c74c251269b88e8ff23e8cd8c4d04477e7d3c79cb47eb6e125943f147a190d0"
Oct 30 13:15:11.724290 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:11.724223    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr" podStartSLOduration=3.239877161 podStartE2EDuration="6.724205404s" totalImagesPullingTime="3.484328243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:11.719557185 +0000 UTC m=+1753.302986903" watchObservedRunningTime="2026-10-30 13:15:11.724205404 +0000 UTC m=+1753.307635098"
Oct 30 13:15:12.500584 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:12.500547    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl" event={"ID":"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4","Type":"ContainerStarted","Data":"63aa4ef88c5fbc0974f032cce8e958eea6ae3e186a66c3c7eb8e38c446a6a9a1"}
Oct 30 13:15:12.502741 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:12.502710    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm" event={"ID":"9dc8baef-b3fd-410f-a85c-60006c08c3ad","Type":"ContainerStarted","Data":"e55a926623427f54187383f339c7bc7bc05e82239b7387f34518040720f34de6"}
Oct 30 13:15:12.505475 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:12.505432    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt" event={"ID":"926867a1-95ec-4376-98ab-c570de7f46bc","Type":"ContainerStarted","Data":"c02b295d27eeca75b39cacd3eb9935c05b75da5b18c661c622c2f7d63f048a2d"}
Oct 30 13:15:12.509305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:12.508789    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp" event={"ID":"21aac6b7-5d00-4753-9159-9c8c31172f95","Type":"ContainerStarted","Data":"b6ecdc8df57058ac1d6328e92a548295b9a863f5993dfcdb39f61571cf9c6942"}
Oct 30 13:15:12.512839 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:12.512809    5893 generic.go:356] "Generic (PLEG): container finished" podID="783977cb-aa63-461f-9fb0-ee490af3989c" containerID="c57a16a9dba0f43d11f1ba6c0f684cdd48e045e44ddd391a40be6bb24653d2fd" exitCode=0
Oct 30 13:15:12.513087 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:12.512885    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr" event={"ID":"783977cb-aa63-461f-9fb0-ee490af3989c","Type":"ContainerDied","Data":"c57a16a9dba0f43d11f1ba6c0f684cdd48e045e44ddd391a40be6bb24653d2fd"}
Oct 30 13:15:12.553750 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:12.553687    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl" podStartSLOduration=3.688305861 podStartE2EDuration="12.553667474s" totalImagesPullingTime="8.865361613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:12.548649044 +0000 UTC m=+1754.132078739" watchObservedRunningTime="2026-10-30 13:15:12.553667474 +0000 UTC m=+1754.137097169"
Oct 30 13:15:12.604626 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:12.604576    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm" podStartSLOduration=3.89752783 podStartE2EDuration="12.604560885s" totalImagesPullingTime="8.707033055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:12.602292804 +0000 UTC m=+1754.185722508" watchObservedRunningTime="2026-10-30 13:15:12.604560885 +0000 UTC m=+1754.187990623"
Oct 30 13:15:12.686453 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:12.686405    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp" podStartSLOduration=3.889540431 podStartE2EDuration="12.686391517s" totalImagesPullingTime="8.796851086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:12.683695525 +0000 UTC m=+1754.267125229" watchObservedRunningTime="2026-10-30 13:15:12.686391517 +0000 UTC m=+1754.269821209"
Oct 30 13:15:13.527153 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.527106    5893 generic.go:356] "Generic (PLEG): container finished" podID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" containerID="63aa4ef88c5fbc0974f032cce8e958eea6ae3e186a66c3c7eb8e38c446a6a9a1" exitCode=60
Oct 30 13:15:13.527701 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.527188    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl" event={"ID":"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4","Type":"ContainerDied","Data":"63aa4ef88c5fbc0974f032cce8e958eea6ae3e186a66c3c7eb8e38c446a6a9a1"}
Oct 30 13:15:13.527701 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.527421    5893 scope.go:122] "RemoveContainer" containerID="63aa4ef88c5fbc0974f032cce8e958eea6ae3e186a66c3c7eb8e38c446a6a9a1"
Oct 30 13:15:13.527701 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.527551    5893 scope.go:122] "RemoveContainer" containerID="9e9c9615b5800d0677427becddf39a04a8e4b9917ffbe849b25a473f4ac9fe10"
Oct 30 13:15:13.527701 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:13.527686    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl_odh-ai-gateway-infra(5c6e949e-0b68-4c8e-bac6-3fccbf584cc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl" podUID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4"
Oct 30 13:15:13.531769 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.531743    5893 generic.go:356] "Generic (PLEG): container finished" podID="9dc8baef-b3fd-410f-a85c-60006c08c3ad" containerID="e55a926623427f54187383f339c7bc7bc05e82239b7387f34518040720f34de6" exitCode=60
Oct 30 13:15:13.531895 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.531833    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm" event={"ID":"9dc8baef-b3fd-410f-a85c-60006c08c3ad","Type":"ContainerDied","Data":"e55a926623427f54187383f339c7bc7bc05e82239b7387f34518040720f34de6"}
Oct 30 13:15:13.532076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.532054    5893 scope.go:122] "RemoveContainer" containerID="e55a926623427f54187383f339c7bc7bc05e82239b7387f34518040720f34de6"
Oct 30 13:15:13.532408 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:13.532384    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm_odh-ai-gateway-infra(9dc8baef-b3fd-410f-a85c-60006c08c3ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm" podUID="9dc8baef-b3fd-410f-a85c-60006c08c3ad"
Oct 30 13:15:13.534406 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.534379    5893 generic.go:356] "Generic (PLEG): container finished" podID="926867a1-95ec-4376-98ab-c570de7f46bc" containerID="c02b295d27eeca75b39cacd3eb9935c05b75da5b18c661c622c2f7d63f048a2d" exitCode=60
Oct 30 13:15:13.534519 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.534495    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt" event={"ID":"926867a1-95ec-4376-98ab-c570de7f46bc","Type":"ContainerDied","Data":"c02b295d27eeca75b39cacd3eb9935c05b75da5b18c661c622c2f7d63f048a2d"}
Oct 30 13:15:13.535287 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.535253    5893 scope.go:122] "RemoveContainer" containerID="c02b295d27eeca75b39cacd3eb9935c05b75da5b18c661c622c2f7d63f048a2d"
Oct 30 13:15:13.535593 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:13.535567    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt_odh-ai-gateway-infra(926867a1-95ec-4376-98ab-c570de7f46bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt" podUID="926867a1-95ec-4376-98ab-c570de7f46bc"
Oct 30 13:15:13.536745 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.536722    5893 generic.go:356] "Generic (PLEG): container finished" podID="21aac6b7-5d00-4753-9159-9c8c31172f95" containerID="b6ecdc8df57058ac1d6328e92a548295b9a863f5993dfcdb39f61571cf9c6942" exitCode=60
Oct 30 13:15:13.536855 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.536806    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp" event={"ID":"21aac6b7-5d00-4753-9159-9c8c31172f95","Type":"ContainerDied","Data":"b6ecdc8df57058ac1d6328e92a548295b9a863f5993dfcdb39f61571cf9c6942"}
Oct 30 13:15:13.537049 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.537030    5893 scope.go:122] "RemoveContainer" containerID="b6ecdc8df57058ac1d6328e92a548295b9a863f5993dfcdb39f61571cf9c6942"
Oct 30 13:15:13.537337 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:13.537314    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp_odh-ai-gateway-infra(21aac6b7-5d00-4753-9159-9c8c31172f95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp" podUID="21aac6b7-5d00-4753-9159-9c8c31172f95"
Oct 30 13:15:13.544399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.544379    5893 scope.go:122] "RemoveContainer" containerID="4c74c251269b88e8ff23e8cd8c4d04477e7d3c79cb47eb6e125943f147a190d0"
Oct 30 13:15:13.559918 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.559888    5893 scope.go:122] "RemoveContainer" containerID="9563a36321a0e0c7cef8ccd47a64c7a4dbc23bf2645720d9203d476397cd5336"
Oct 30 13:15:13.582978 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.582958    5893 scope.go:122] "RemoveContainer" containerID="cdaadf4d599a929e5558c18d9ce55aedadd4439e6a593faa4aefd99becdec134"
Oct 30 13:15:13.770556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.770526    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"
Oct 30 13:15:13.890844 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.890768    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/783977cb-aa63-461f-9fb0-ee490af3989c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783977cb-aa63-461f-9fb0-ee490af3989c-maas-api-xks-ca\") pod \"783977cb-aa63-461f-9fb0-ee490af3989c\" (UID: \"783977cb-aa63-461f-9fb0-ee490af3989c\") "
Oct 30 13:15:13.891004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.890884    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/783977cb-aa63-461f-9fb0-ee490af3989c-kube-api-access-xxxwx\" (UniqueName: \"kubernetes.io/projected/783977cb-aa63-461f-9fb0-ee490af3989c-kube-api-access-xxxwx\") pod \"783977cb-aa63-461f-9fb0-ee490af3989c\" (UID: \"783977cb-aa63-461f-9fb0-ee490af3989c\") "
Oct 30 13:15:13.891051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.891033    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/783977cb-aa63-461f-9fb0-ee490af3989c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783977cb-aa63-461f-9fb0-ee490af3989c-maas-api-openshift-ca\") pod \"783977cb-aa63-461f-9fb0-ee490af3989c\" (UID: \"783977cb-aa63-461f-9fb0-ee490af3989c\") "
Oct 30 13:15:13.891393 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.891368    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/783977cb-aa63-461f-9fb0-ee490af3989c-maas-api-openshift-ca" pod "783977cb-aa63-461f-9fb0-ee490af3989c" (UID: "783977cb-aa63-461f-9fb0-ee490af3989c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:13.891700 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.891680    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/783977cb-aa63-461f-9fb0-ee490af3989c-maas-api-openshift-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.893657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.893633    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/783977cb-aa63-461f-9fb0-ee490af3989c-maas-api-xks-ca" pod "783977cb-aa63-461f-9fb0-ee490af3989c" (UID: "783977cb-aa63-461f-9fb0-ee490af3989c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:15:13.893740 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.893678    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/783977cb-aa63-461f-9fb0-ee490af3989c-kube-api-access-xxxwx" pod "783977cb-aa63-461f-9fb0-ee490af3989c" (UID: "783977cb-aa63-461f-9fb0-ee490af3989c"). InnerVolumeSpecName "kube-api-access-xxxwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:13.993381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.993349    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/783977cb-aa63-461f-9fb0-ee490af3989c-maas-api-xks-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:13.993381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:13.993374    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxxwx\" (UniqueName: \"kubernetes.io/projected/783977cb-aa63-461f-9fb0-ee490af3989c-kube-api-access-xxxwx\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:14.547233 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:14.547207    5893 scope.go:122] "RemoveContainer" containerID="c02b295d27eeca75b39cacd3eb9935c05b75da5b18c661c622c2f7d63f048a2d"
Oct 30 13:15:14.547652 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:14.547626    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt_odh-ai-gateway-infra(926867a1-95ec-4376-98ab-c570de7f46bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt" podUID="926867a1-95ec-4376-98ab-c570de7f46bc"
Oct 30 13:15:14.548996 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:14.548973    5893 scope.go:122] "RemoveContainer" containerID="b6ecdc8df57058ac1d6328e92a548295b9a863f5993dfcdb39f61571cf9c6942"
Oct 30 13:15:14.549344 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:14.549322    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp_odh-ai-gateway-infra(21aac6b7-5d00-4753-9159-9c8c31172f95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp" podUID="21aac6b7-5d00-4753-9159-9c8c31172f95"
Oct 30 13:15:14.550115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:14.550100    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"
Oct 30 13:15:14.550219 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:14.550158    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr" event={"ID":"783977cb-aa63-461f-9fb0-ee490af3989c","Type":"ContainerDied","Data":"0b1b290598827f70970a5aefeebea5669881fbbf208b4335599d107018268bb2"}
Oct 30 13:15:14.550219 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:14.550207    5893 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b1b290598827f70970a5aefeebea5669881fbbf208b4335599d107018268bb2"
Oct 30 13:15:14.552436 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:14.552420    5893 scope.go:122] "RemoveContainer" containerID="63aa4ef88c5fbc0974f032cce8e958eea6ae3e186a66c3c7eb8e38c446a6a9a1"
Oct 30 13:15:14.552824 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:14.552799    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl_odh-ai-gateway-infra(5c6e949e-0b68-4c8e-bac6-3fccbf584cc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl" podUID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4"
Oct 30 13:15:14.554389 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:14.554372    5893 scope.go:122] "RemoveContainer" containerID="e55a926623427f54187383f339c7bc7bc05e82239b7387f34518040720f34de6"
Oct 30 13:15:14.554590 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:14.554573    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm_odh-ai-gateway-infra(9dc8baef-b3fd-410f-a85c-60006c08c3ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm" podUID="9dc8baef-b3fd-410f-a85c-60006c08c3ad"
Oct 30 13:15:16.325255 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:16.325217    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"]
Oct 30 13:15:16.369591 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:16.369557    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"]
Oct 30 13:15:18.658390 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:18.652740    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:15:19.064760 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:19.064072    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"]
Oct 30 13:15:20.474743 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:20.474715    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"
Oct 30 13:15:21.325561 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.325523    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl" podUID="0769c09e-fccd-4411-9a8f-824acd1dcd27" containerName="payload-processing" containerID="cri-o://cd1335ba203affc8174f23dee6d48359c599728355ab7e5d243365955a86ff8c" gracePeriod=25
Oct 30 13:15:21.370411 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.370369    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz" podUID="d261ff50-28dc-4622-8dae-75501cf12028" containerName="payload-pre-processing" containerID="cri-o://b899adcbf99a135deb171b52fee02722eff5c2b8d3750e3cfa9d87a68d321104" gracePeriod=25
Oct 30 13:15:21.495946 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.495920    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"
Oct 30 13:15:21.648768 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.648729    5893 generic.go:356] "Generic (PLEG): container finished" podID="d261ff50-28dc-4622-8dae-75501cf12028" containerID="b899adcbf99a135deb171b52fee02722eff5c2b8d3750e3cfa9d87a68d321104" exitCode=0
Oct 30 13:15:21.649019 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.648992    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz" event={"ID":"d261ff50-28dc-4622-8dae-75501cf12028","Type":"ContainerDied","Data":"b899adcbf99a135deb171b52fee02722eff5c2b8d3750e3cfa9d87a68d321104"}
Oct 30 13:15:21.651492 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.651343    5893 generic.go:356] "Generic (PLEG): container finished" podID="0769c09e-fccd-4411-9a8f-824acd1dcd27" containerID="cd1335ba203affc8174f23dee6d48359c599728355ab7e5d243365955a86ff8c" exitCode=0
Oct 30 13:15:21.651492 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.651437    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl" event={"ID":"0769c09e-fccd-4411-9a8f-824acd1dcd27","Type":"ContainerDied","Data":"cd1335ba203affc8174f23dee6d48359c599728355ab7e5d243365955a86ff8c"}
Oct 30 13:15:21.695638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.695158    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"
Oct 30 13:15:21.813137 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.812802    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"
Oct 30 13:15:21.840050 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.840020    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0769c09e-fccd-4411-9a8f-824acd1dcd27-kube-api-access-6hxg6\" (UniqueName: \"kubernetes.io/projected/0769c09e-fccd-4411-9a8f-824acd1dcd27-kube-api-access-6hxg6\") pod \"0769c09e-fccd-4411-9a8f-824acd1dcd27\" (UID: \"0769c09e-fccd-4411-9a8f-824acd1dcd27\") "
Oct 30 13:15:21.840247 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.840203    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0769c09e-fccd-4411-9a8f-824acd1dcd27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0769c09e-fccd-4411-9a8f-824acd1dcd27-plugins-config-volume\") pod \"0769c09e-fccd-4411-9a8f-824acd1dcd27\" (UID: \"0769c09e-fccd-4411-9a8f-824acd1dcd27\") "
Oct 30 13:15:21.841114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.841089    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0769c09e-fccd-4411-9a8f-824acd1dcd27-plugins-config-volume" pod "0769c09e-fccd-4411-9a8f-824acd1dcd27" (UID: "0769c09e-fccd-4411-9a8f-824acd1dcd27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:21.848412 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.848391    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0769c09e-fccd-4411-9a8f-824acd1dcd27-kube-api-access-6hxg6" pod "0769c09e-fccd-4411-9a8f-824acd1dcd27" (UID: "0769c09e-fccd-4411-9a8f-824acd1dcd27"). InnerVolumeSpecName "kube-api-access-6hxg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:21.942673 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.942581    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d261ff50-28dc-4622-8dae-75501cf12028-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d261ff50-28dc-4622-8dae-75501cf12028-plugins-config-volume\") pod \"d261ff50-28dc-4622-8dae-75501cf12028\" (UID: \"d261ff50-28dc-4622-8dae-75501cf12028\") "
Oct 30 13:15:21.942910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.942716    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d261ff50-28dc-4622-8dae-75501cf12028-kube-api-access-czsgf\" (UniqueName: \"kubernetes.io/projected/d261ff50-28dc-4622-8dae-75501cf12028-kube-api-access-czsgf\") pod \"d261ff50-28dc-4622-8dae-75501cf12028\" (UID: \"d261ff50-28dc-4622-8dae-75501cf12028\") "
Oct 30 13:15:21.943869 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.943840    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hxg6\" (UniqueName: \"kubernetes.io/projected/0769c09e-fccd-4411-9a8f-824acd1dcd27-kube-api-access-6hxg6\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:21.944047 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.943876    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0769c09e-fccd-4411-9a8f-824acd1dcd27-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:21.944461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.944410    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d261ff50-28dc-4622-8dae-75501cf12028-plugins-config-volume" pod "d261ff50-28dc-4622-8dae-75501cf12028" (UID: "d261ff50-28dc-4622-8dae-75501cf12028"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:21.947201 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:21.947169    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d261ff50-28dc-4622-8dae-75501cf12028-kube-api-access-czsgf" pod "d261ff50-28dc-4622-8dae-75501cf12028" (UID: "d261ff50-28dc-4622-8dae-75501cf12028"). InnerVolumeSpecName "kube-api-access-czsgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:22.048480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:22.046199    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d261ff50-28dc-4622-8dae-75501cf12028-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.048480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:22.046242    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czsgf\" (UniqueName: \"kubernetes.io/projected/d261ff50-28dc-4622-8dae-75501cf12028-kube-api-access-czsgf\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:22.660932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:22.660852    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"
Oct 30 13:15:22.660932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:22.660881    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl" event={"ID":"0769c09e-fccd-4411-9a8f-824acd1dcd27","Type":"ContainerDied","Data":"37fc5e331ab019e64c6142d8a2b9890512522169daca6492a198bc2917f462f4"}
Oct 30 13:15:22.661448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:22.660952    5893 scope.go:122] "RemoveContainer" containerID="cd1335ba203affc8174f23dee6d48359c599728355ab7e5d243365955a86ff8c"
Oct 30 13:15:22.662454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:22.662434    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz" event={"ID":"d261ff50-28dc-4622-8dae-75501cf12028","Type":"ContainerDied","Data":"ff954e22285f7aea89ea6fcfdc700d2860948a46c503d3d42e846ffbefc9b340"}
Oct 30 13:15:22.662574 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:22.662451    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"
Oct 30 13:15:22.671799 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:22.671781    5893 scope.go:122] "RemoveContainer" containerID="b899adcbf99a135deb171b52fee02722eff5c2b8d3750e3cfa9d87a68d321104"
Oct 30 13:15:22.722206 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:22.722177    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"]
Oct 30 13:15:22.733617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:22.733594    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-abbc6c-6884d59d7f-xn4kz"]
Oct 30 13:15:22.772371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:22.772348    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"]
Oct 30 13:15:22.775646 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:22.775628    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-abbc6c-64bbd756cb-qhdcl"]
Oct 30 13:15:23.523920 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:23.523891    5893 scope.go:122] "RemoveContainer" containerID="c02b295d27eeca75b39cacd3eb9935c05b75da5b18c661c622c2f7d63f048a2d"
Oct 30 13:15:24.064891 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.064860    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"]
Oct 30 13:15:24.064891 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.064878    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q" podUID="a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013" containerName="payload-processing" containerID="cri-o://11607827ab87fca40c1caaf805435fb971b0194f63590f554ae731c09f4ae64f" gracePeriod=25
Oct 30 13:15:24.065767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.065738    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d261ff50-28dc-4622-8dae-75501cf12028" containerName="payload-pre-processing"
Oct 30 13:15:24.065767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.065758    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d261ff50-28dc-4622-8dae-75501cf12028" containerName="payload-pre-processing"
Oct 30 13:15:24.065938 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.065783    5893 cpu_manager.go:414] "containerMap: removing container" podUID="0769c09e-fccd-4411-9a8f-824acd1dcd27" containerName="payload-processing"
Oct 30 13:15:24.065938 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.065789    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0769c09e-fccd-4411-9a8f-824acd1dcd27" containerName="payload-processing"
Oct 30 13:15:24.065938 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.065802    5893 cpu_manager.go:414] "containerMap: removing container" podUID="783977cb-aa63-461f-9fb0-ee490af3989c" containerName="revoke-keys"
Oct 30 13:15:24.065938 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.065807    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="783977cb-aa63-461f-9fb0-ee490af3989c" containerName="revoke-keys"
Oct 30 13:15:24.066084 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.065973    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d261ff50-28dc-4622-8dae-75501cf12028" containerName="payload-pre-processing"
Oct 30 13:15:24.066084 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.065987    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="0769c09e-fccd-4411-9a8f-824acd1dcd27" containerName="payload-processing"
Oct 30 13:15:24.066084 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.065993    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="783977cb-aa63-461f-9fb0-ee490af3989c" containerName="revoke-keys"
Oct 30 13:15:24.069830 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.069810    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:24.094972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.094942    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-649303-serving-cert\""
Oct 30 13:15:24.103908 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.103817    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"]
Oct 30 13:15:24.175774 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.175731    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c80976-1d79-48e5-8718-2695ae37d011-openshift-service-ca\") pod \"maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:24.175961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.175900    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjnnk\" (UniqueName: \"kubernetes.io/projected/62c80976-1d79-48e5-8718-2695ae37d011-kube-api-access-mjnnk\") pod \"maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:24.176007 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.175986    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c80976-1d79-48e5-8718-2695ae37d011-metrics-certs\") pod \"maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:24.176115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.176094    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62c80976-1d79-48e5-8718-2695ae37d011-maas-api-tls\") pod \"maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:24.278458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.278421    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c80976-1d79-48e5-8718-2695ae37d011-openshift-service-ca\") pod \"maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:24.278644 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.278543    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjnnk\" (UniqueName: \"kubernetes.io/projected/62c80976-1d79-48e5-8718-2695ae37d011-kube-api-access-mjnnk\") pod \"maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:24.278742 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.278647    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c80976-1d79-48e5-8718-2695ae37d011-metrics-certs\") pod \"maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:24.278835 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.278745    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62c80976-1d79-48e5-8718-2695ae37d011-maas-api-tls\") pod \"maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:24.280382 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.279262    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c80976-1d79-48e5-8718-2695ae37d011-openshift-service-ca\") pod \"maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:24.283370 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.281790    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62c80976-1d79-48e5-8718-2695ae37d011-maas-api-tls\") pod \"maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:24.283370 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.281894    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c80976-1d79-48e5-8718-2695ae37d011-metrics-certs\") pod \"maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:24.303399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.303366    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjnnk\" (UniqueName: \"kubernetes.io/projected/62c80976-1d79-48e5-8718-2695ae37d011-kube-api-access-mjnnk\") pod \"maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:24.387128 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.387099    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"
Oct 30 13:15:24.442170 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.442137    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:24.482605 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.482568    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013-plugins-config-volume\") pod \"a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013\" (UID: \"a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013\") "
Oct 30 13:15:24.483040 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.483018    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013-plugins-config-volume" pod "a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013" (UID: "a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:24.483131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.483114    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013-kube-api-access-k68dr\" (UniqueName: \"kubernetes.io/projected/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013-kube-api-access-k68dr\") pod \"a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013\" (UID: \"a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013\") "
Oct 30 13:15:24.483910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.483889    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:24.486550 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.486525    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013-kube-api-access-k68dr" pod "a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013" (UID: "a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013"). InnerVolumeSpecName "kube-api-access-k68dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:24.527380 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.524566    5893 scope.go:122] "RemoveContainer" containerID="b6ecdc8df57058ac1d6328e92a548295b9a863f5993dfcdb39f61571cf9c6942"
Oct 30 13:15:24.527380 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.524818    5893 scope.go:122] "RemoveContainer" containerID="63aa4ef88c5fbc0974f032cce8e958eea6ae3e186a66c3c7eb8e38c446a6a9a1"
Oct 30 13:15:24.527380 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.525697    5893 scope.go:122] "RemoveContainer" containerID="e55a926623427f54187383f339c7bc7bc05e82239b7387f34518040720f34de6"
Oct 30 13:15:24.555287 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.555237    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0769c09e-fccd-4411-9a8f-824acd1dcd27" path="/var/lib/kubelet/pods/0769c09e-fccd-4411-9a8f-824acd1dcd27/volumes"
Oct 30 13:15:24.556297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.556255    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d261ff50-28dc-4622-8dae-75501cf12028" path="/var/lib/kubelet/pods/d261ff50-28dc-4622-8dae-75501cf12028/volumes"
Oct 30 13:15:24.586689 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.586657    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k68dr\" (UniqueName: \"kubernetes.io/projected/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013-kube-api-access-k68dr\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:24.687774 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.687728    5893 generic.go:356] "Generic (PLEG): container finished" podID="a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013" containerID="11607827ab87fca40c1caaf805435fb971b0194f63590f554ae731c09f4ae64f" exitCode=0
Oct 30 13:15:24.688034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.688003    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q" event={"ID":"a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013","Type":"ContainerDied","Data":"11607827ab87fca40c1caaf805435fb971b0194f63590f554ae731c09f4ae64f"}
Oct 30 13:15:24.688127 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.688113    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q" event={"ID":"a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013","Type":"ContainerDied","Data":"ab0608b2bab8a998ba7d25a23f95076c08bae0dad9f900020b524c895fe09c03"}
Oct 30 13:15:24.688194 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.688180    5893 scope.go:122] "RemoveContainer" containerID="11607827ab87fca40c1caaf805435fb971b0194f63590f554ae731c09f4ae64f"
Oct 30 13:15:24.688413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.688397    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"
Oct 30 13:15:24.700606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.700577    5893 generic.go:356] "Generic (PLEG): container finished" podID="926867a1-95ec-4376-98ab-c570de7f46bc" containerID="e644c428168b37bd51f87784680ce066632a270cedd204a78ce239455949c9b5" exitCode=60
Oct 30 13:15:24.700755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.700669    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt" event={"ID":"926867a1-95ec-4376-98ab-c570de7f46bc","Type":"ContainerDied","Data":"e644c428168b37bd51f87784680ce066632a270cedd204a78ce239455949c9b5"}
Oct 30 13:15:24.701220 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.701151    5893 scope.go:122] "RemoveContainer" containerID="e644c428168b37bd51f87784680ce066632a270cedd204a78ce239455949c9b5"
Oct 30 13:15:24.701580 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:24.701555    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt_odh-ai-gateway-infra(926867a1-95ec-4376-98ab-c570de7f46bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt" podUID="926867a1-95ec-4376-98ab-c570de7f46bc"
Oct 30 13:15:24.709773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.709750    5893 scope.go:122] "RemoveContainer" containerID="11607827ab87fca40c1caaf805435fb971b0194f63590f554ae731c09f4ae64f"
Oct 30 13:15:24.710094 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:24.710073    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11607827ab87fca40c1caaf805435fb971b0194f63590f554ae731c09f4ae64f\": container with ID starting with 11607827ab87fca40c1caaf805435fb971b0194f63590f554ae731c09f4ae64f not found: ID does not exist" containerID="11607827ab87fca40c1caaf805435fb971b0194f63590f554ae731c09f4ae64f"
Oct 30 13:15:24.710180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.710101    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11607827ab87fca40c1caaf805435fb971b0194f63590f554ae731c09f4ae64f"} err="failed to get container status \"11607827ab87fca40c1caaf805435fb971b0194f63590f554ae731c09f4ae64f\": rpc error: code = NotFound desc = could not find container \"11607827ab87fca40c1caaf805435fb971b0194f63590f554ae731c09f4ae64f\": container with ID starting with 11607827ab87fca40c1caaf805435fb971b0194f63590f554ae731c09f4ae64f not found: ID does not exist"
Oct 30 13:15:24.710180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.710119    5893 scope.go:122] "RemoveContainer" containerID="c02b295d27eeca75b39cacd3eb9935c05b75da5b18c661c622c2f7d63f048a2d"
Oct 30 13:15:24.785100 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.785070    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"]
Oct 30 13:15:24.797357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.797325    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-716b4c-85c6b65cb6-4fw9q"]
Oct 30 13:15:24.814081 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:24.813963    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"]
Oct 30 13:15:24.818414 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:15:24.818378    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c80976_1d79_48e5_8718_2695ae37d011.slice/crio-c3b74d5843f768d5848130192e8a35a36baf65cd8ef3862d54c1c5b1e4416d1f WatchSource:0}: Error finding container c3b74d5843f768d5848130192e8a35a36baf65cd8ef3862d54c1c5b1e4416d1f: Status 404 returned error can't find the container with id c3b74d5843f768d5848130192e8a35a36baf65cd8ef3862d54c1c5b1e4416d1f
Oct 30 13:15:25.712047 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:25.712012    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp" event={"ID":"21aac6b7-5d00-4753-9159-9c8c31172f95","Type":"ContainerStarted","Data":"b9b7f202d76564f329e4717e46614d2db12a29bf04fbecd85abec209fff1208b"}
Oct 30 13:15:25.714038 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:25.714012    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl" event={"ID":"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4","Type":"ContainerStarted","Data":"25558aab48844b547006a246651c78032a72357fae331bf41c5035c84b24746b"}
Oct 30 13:15:25.715591 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:25.715571    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6" event={"ID":"62c80976-1d79-48e5-8718-2695ae37d011","Type":"ContainerStarted","Data":"365d2a330d543c11bd861b6c03ff496a23a92c322e10e9711a085c2f398c438f"}
Oct 30 13:15:25.715702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:25.715615    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6" event={"ID":"62c80976-1d79-48e5-8718-2695ae37d011","Type":"ContainerStarted","Data":"c3b74d5843f768d5848130192e8a35a36baf65cd8ef3862d54c1c5b1e4416d1f"}
Oct 30 13:15:25.715702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:25.715647    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:25.717302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:25.717265    5893 generic.go:356] "Generic (PLEG): container finished" podID="9dc8baef-b3fd-410f-a85c-60006c08c3ad" containerID="f6724f93399c0d9bfe7f22324233324c0ec285f9451bf008c3cc7c26b410f6da" exitCode=60
Oct 30 13:15:25.717405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:25.717307    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm" event={"ID":"9dc8baef-b3fd-410f-a85c-60006c08c3ad","Type":"ContainerDied","Data":"f6724f93399c0d9bfe7f22324233324c0ec285f9451bf008c3cc7c26b410f6da"}
Oct 30 13:15:25.717405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:25.717353    5893 scope.go:122] "RemoveContainer" containerID="e55a926623427f54187383f339c7bc7bc05e82239b7387f34518040720f34de6"
Oct 30 13:15:25.717672 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:25.717657    5893 scope.go:122] "RemoveContainer" containerID="f6724f93399c0d9bfe7f22324233324c0ec285f9451bf008c3cc7c26b410f6da"
Oct 30 13:15:25.717987 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:25.717968    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm_odh-ai-gateway-infra(9dc8baef-b3fd-410f-a85c-60006c08c3ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm" podUID="9dc8baef-b3fd-410f-a85c-60006c08c3ad"
Oct 30 13:15:25.764660 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:25.764628    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"]
Oct 30 13:15:25.802933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:25.802318    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6" podStartSLOduration=1.474155838 podStartE2EDuration="1.802253097s" totalImagesPullingTime="328.097259ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:25.801596574 +0000 UTC m=+1767.385026273" watchObservedRunningTime="2026-10-30 13:15:25.802253097 +0000 UTC m=+1767.385682797"
Oct 30 13:15:25.956643 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:25.956616    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"
Oct 30 13:15:26.126394 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.126357    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/926867a1-95ec-4376-98ab-c570de7f46bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926867a1-95ec-4376-98ab-c570de7f46bc-maas-api-service-ca\") pod \"926867a1-95ec-4376-98ab-c570de7f46bc\" (UID: \"926867a1-95ec-4376-98ab-c570de7f46bc\") "
Oct 30 13:15:26.126555 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.126506    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/926867a1-95ec-4376-98ab-c570de7f46bc-kube-api-access-8grwj\" (UniqueName: \"kubernetes.io/projected/926867a1-95ec-4376-98ab-c570de7f46bc-kube-api-access-8grwj\") pod \"926867a1-95ec-4376-98ab-c570de7f46bc\" (UID: \"926867a1-95ec-4376-98ab-c570de7f46bc\") "
Oct 30 13:15:26.126795 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.126773    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/926867a1-95ec-4376-98ab-c570de7f46bc-maas-api-service-ca" pod "926867a1-95ec-4376-98ab-c570de7f46bc" (UID: "926867a1-95ec-4376-98ab-c570de7f46bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:26.127112 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.127097    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/926867a1-95ec-4376-98ab-c570de7f46bc-maas-api-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:26.128762 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.128742    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/926867a1-95ec-4376-98ab-c570de7f46bc-kube-api-access-8grwj" pod "926867a1-95ec-4376-98ab-c570de7f46bc" (UID: "926867a1-95ec-4376-98ab-c570de7f46bc"). InnerVolumeSpecName "kube-api-access-8grwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:26.228607 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.228577    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8grwj\" (UniqueName: \"kubernetes.io/projected/926867a1-95ec-4376-98ab-c570de7f46bc-kube-api-access-8grwj\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:26.533211 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.533134    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013" path="/var/lib/kubelet/pods/a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013/volumes"
Oct 30 13:15:26.751649 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.751616    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"
Oct 30 13:15:26.753234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.753205    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt" event={"ID":"926867a1-95ec-4376-98ab-c570de7f46bc","Type":"ContainerDied","Data":"0725755f94299a2f41b55a6044bd8cde1af2ef6986d61860308d9a4f857b2025"}
Oct 30 13:15:26.755172 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.755148    5893 scope.go:122] "RemoveContainer" containerID="e644c428168b37bd51f87784680ce066632a270cedd204a78ce239455949c9b5"
Oct 30 13:15:26.759116 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.759081    5893 generic.go:356] "Generic (PLEG): container finished" podID="21aac6b7-5d00-4753-9159-9c8c31172f95" containerID="b9b7f202d76564f329e4717e46614d2db12a29bf04fbecd85abec209fff1208b" exitCode=60
Oct 30 13:15:26.759231 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.759187    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp" event={"ID":"21aac6b7-5d00-4753-9159-9c8c31172f95","Type":"ContainerDied","Data":"b9b7f202d76564f329e4717e46614d2db12a29bf04fbecd85abec209fff1208b"}
Oct 30 13:15:26.759751 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.759728    5893 scope.go:122] "RemoveContainer" containerID="b9b7f202d76564f329e4717e46614d2db12a29bf04fbecd85abec209fff1208b"
Oct 30 13:15:26.760150 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:26.760122    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp_odh-ai-gateway-infra(21aac6b7-5d00-4753-9159-9c8c31172f95)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp" podUID="21aac6b7-5d00-4753-9159-9c8c31172f95"
Oct 30 13:15:26.761847 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.761821    5893 generic.go:356] "Generic (PLEG): container finished" podID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" containerID="25558aab48844b547006a246651c78032a72357fae331bf41c5035c84b24746b" exitCode=60
Oct 30 13:15:26.761970 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.761945    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl" event={"ID":"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4","Type":"ContainerDied","Data":"25558aab48844b547006a246651c78032a72357fae331bf41c5035c84b24746b"}
Oct 30 13:15:26.762306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.762255    5893 scope.go:122] "RemoveContainer" containerID="25558aab48844b547006a246651c78032a72357fae331bf41c5035c84b24746b"
Oct 30 13:15:26.762613 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:26.762594    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl_odh-ai-gateway-infra(5c6e949e-0b68-4c8e-bac6-3fccbf584cc4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl" podUID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4"
Oct 30 13:15:26.772551 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.772531    5893 scope.go:122] "RemoveContainer" containerID="b6ecdc8df57058ac1d6328e92a548295b9a863f5993dfcdb39f61571cf9c6942"
Oct 30 13:15:26.772659 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.772646    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"]
Oct 30 13:15:26.790567 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.790544    5893 scope.go:122] "RemoveContainer" containerID="63aa4ef88c5fbc0974f032cce8e958eea6ae3e186a66c3c7eb8e38c446a6a9a1"
Oct 30 13:15:26.812728 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.812699    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"]
Oct 30 13:15:26.816023 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.816002    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-2afe58-29889435-gn9pt"]
Oct 30 13:15:26.861714 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.861687    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"]
Oct 30 13:15:26.902286 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:26.902236    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"]
Oct 30 13:15:27.064158 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:27.064090    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"
Oct 30 13:15:27.259319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:27.259264    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dc8baef-b3fd-410f-a85c-60006c08c3ad-kube-api-access-gvpll\" (UniqueName: \"kubernetes.io/projected/9dc8baef-b3fd-410f-a85c-60006c08c3ad-kube-api-access-gvpll\") pod \"9dc8baef-b3fd-410f-a85c-60006c08c3ad\" (UID: \"9dc8baef-b3fd-410f-a85c-60006c08c3ad\") "
Oct 30 13:15:27.259529 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:27.259377    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dc8baef-b3fd-410f-a85c-60006c08c3ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8baef-b3fd-410f-a85c-60006c08c3ad-maas-api-service-ca\") pod \"9dc8baef-b3fd-410f-a85c-60006c08c3ad\" (UID: \"9dc8baef-b3fd-410f-a85c-60006c08c3ad\") "
Oct 30 13:15:27.259787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:27.259762    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dc8baef-b3fd-410f-a85c-60006c08c3ad-maas-api-service-ca" pod "9dc8baef-b3fd-410f-a85c-60006c08c3ad" (UID: "9dc8baef-b3fd-410f-a85c-60006c08c3ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:27.260150 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:27.260131    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9dc8baef-b3fd-410f-a85c-60006c08c3ad-maas-api-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:27.261649 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:27.261622    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dc8baef-b3fd-410f-a85c-60006c08c3ad-kube-api-access-gvpll" pod "9dc8baef-b3fd-410f-a85c-60006c08c3ad" (UID: "9dc8baef-b3fd-410f-a85c-60006c08c3ad"). InnerVolumeSpecName "kube-api-access-gvpll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:27.361514 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:27.361426    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvpll\" (UniqueName: \"kubernetes.io/projected/9dc8baef-b3fd-410f-a85c-60006c08c3ad-kube-api-access-gvpll\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:27.792252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:27.792223    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm" event={"ID":"9dc8baef-b3fd-410f-a85c-60006c08c3ad","Type":"ContainerDied","Data":"1b81fcbc8598a7c1d5ca42230661c5a093c5cd4c0a32df64b9eb5e8566d776f5"}
Oct 30 13:15:27.792618 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:27.792348    5893 scope.go:122] "RemoveContainer" containerID="f6724f93399c0d9bfe7f22324233324c0ec285f9451bf008c3cc7c26b410f6da"
Oct 30 13:15:27.792618 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:27.792439    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"
Oct 30 13:15:27.903761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:27.903686    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"]
Oct 30 13:15:27.950386 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:27.950355    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-acd125-29889435-44nnm"]
Oct 30 13:15:28.002795 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.002770    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"
Oct 30 13:15:28.035881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.035852    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"
Oct 30 13:15:28.086902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.086829    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21aac6b7-5d00-4753-9159-9c8c31172f95-kube-api-access-5d86n\" (UniqueName: \"kubernetes.io/projected/21aac6b7-5d00-4753-9159-9c8c31172f95-kube-api-access-5d86n\") pod \"21aac6b7-5d00-4753-9159-9c8c31172f95\" (UID: \"21aac6b7-5d00-4753-9159-9c8c31172f95\") "
Oct 30 13:15:28.086902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.086896    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4-kube-api-access-qf9wc\" (UniqueName: \"kubernetes.io/projected/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4-kube-api-access-qf9wc\") pod \"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4\" (UID: \"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4\") "
Oct 30 13:15:28.087119 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.086948    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4-maas-api-service-ca\") pod \"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4\" (UID: \"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4\") "
Oct 30 13:15:28.087119 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.087050    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21aac6b7-5d00-4753-9159-9c8c31172f95-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21aac6b7-5d00-4753-9159-9c8c31172f95-maas-api-service-ca\") pod \"21aac6b7-5d00-4753-9159-9c8c31172f95\" (UID: \"21aac6b7-5d00-4753-9159-9c8c31172f95\") "
Oct 30 13:15:28.087362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.087337    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4-maas-api-service-ca" pod "5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" (UID: "5c6e949e-0b68-4c8e-bac6-3fccbf584cc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:28.087481 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.087427    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21aac6b7-5d00-4753-9159-9c8c31172f95-maas-api-service-ca" pod "21aac6b7-5d00-4753-9159-9c8c31172f95" (UID: "21aac6b7-5d00-4753-9159-9c8c31172f95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:28.087739 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.087691    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4-maas-api-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:28.087739 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.087712    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21aac6b7-5d00-4753-9159-9c8c31172f95-maas-api-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:28.089392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.089366    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21aac6b7-5d00-4753-9159-9c8c31172f95-kube-api-access-5d86n" pod "21aac6b7-5d00-4753-9159-9c8c31172f95" (UID: "21aac6b7-5d00-4753-9159-9c8c31172f95"). InnerVolumeSpecName "kube-api-access-5d86n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:28.089496 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.089397    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4-kube-api-access-qf9wc" pod "5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" (UID: "5c6e949e-0b68-4c8e-bac6-3fccbf584cc4"). InnerVolumeSpecName "kube-api-access-qf9wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:28.189246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.189204    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d86n\" (UniqueName: \"kubernetes.io/projected/21aac6b7-5d00-4753-9159-9c8c31172f95-kube-api-access-5d86n\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:28.189246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.189235    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf9wc\" (UniqueName: \"kubernetes.io/projected/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4-kube-api-access-qf9wc\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:28.533343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.533304    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="926867a1-95ec-4376-98ab-c570de7f46bc" path="/var/lib/kubelet/pods/926867a1-95ec-4376-98ab-c570de7f46bc/volumes"
Oct 30 13:15:28.533897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.533875    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dc8baef-b3fd-410f-a85c-60006c08c3ad" path="/var/lib/kubelet/pods/9dc8baef-b3fd-410f-a85c-60006c08c3ad/volumes"
Oct 30 13:15:28.810030 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.809947    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"
Oct 30 13:15:28.810030 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.810004    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp" event={"ID":"21aac6b7-5d00-4753-9159-9c8c31172f95","Type":"ContainerDied","Data":"eacca07bccc5f9c49df1918568002ec2660ed42cd9d54ee089e61038870fa467"}
Oct 30 13:15:28.810565 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.810074    5893 scope.go:122] "RemoveContainer" containerID="b9b7f202d76564f329e4717e46614d2db12a29bf04fbecd85abec209fff1208b"
Oct 30 13:15:28.812046 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.812029    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"
Oct 30 13:15:28.812695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.812470    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl" event={"ID":"5c6e949e-0b68-4c8e-bac6-3fccbf584cc4","Type":"ContainerDied","Data":"09c59c1c028e9912ca66e6bdf165ca69386738346af1f09dd441fba065195c5c"}
Oct 30 13:15:28.826785 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.826759    5893 scope.go:122] "RemoveContainer" containerID="25558aab48844b547006a246651c78032a72357fae331bf41c5035c84b24746b"
Oct 30 13:15:28.873578 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.873552    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"]
Oct 30 13:15:28.883100 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.883074    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-0e874e-29889435-msfzl"]
Oct 30 13:15:28.923537 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.923507    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"]
Oct 30 13:15:28.926899 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:28.926874    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-abbc6c-29889435-qkvlp"]
Oct 30 13:15:30.532637 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:30.532603    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21aac6b7-5d00-4753-9159-9c8c31172f95" path="/var/lib/kubelet/pods/21aac6b7-5d00-4753-9159-9c8c31172f95/volumes"
Oct 30 13:15:30.533077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:30.533063    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" path="/var/lib/kubelet/pods/5c6e949e-0b68-4c8e-bac6-3fccbf584cc4/volumes"
Oct 30 13:15:31.770556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:31.770527    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:15:43.448213 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:43.448171    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"]
Oct 30 13:15:43.448727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:43.448482    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn" podUID="3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" containerName="istio-proxy" containerID="cri-o://cb6e8e7285097ca05a81f20d35e3b89cbb2899d09edad40d05f6067b821a6ca5" gracePeriod=30
Oct 30 13:15:45.518292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.518230    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"]
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521248    5893 cpu_manager.go:414] "containerMap: removing container" podUID="9dc8baef-b3fd-410f-a85c-60006c08c3ad" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521326    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc8baef-b3fd-410f-a85c-60006c08c3ad" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521382    5893 cpu_manager.go:414] "containerMap: removing container" podUID="21aac6b7-5d00-4753-9159-9c8c31172f95" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521392    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21aac6b7-5d00-4753-9159-9c8c31172f95" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521406    5893 cpu_manager.go:414] "containerMap: removing container" podUID="926867a1-95ec-4376-98ab-c570de7f46bc" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521413    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926867a1-95ec-4376-98ab-c570de7f46bc" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521428    5893 cpu_manager.go:414] "containerMap: removing container" podUID="a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013" containerName="payload-processing"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521435    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013" containerName="payload-processing"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521476    5893 cpu_manager.go:414] "containerMap: removing container" podUID="9dc8baef-b3fd-410f-a85c-60006c08c3ad" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521483    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc8baef-b3fd-410f-a85c-60006c08c3ad" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521540    5893 cpu_manager.go:414] "containerMap: removing container" podUID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521548    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521579    5893 cpu_manager.go:414] "containerMap: removing container" podUID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521586    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521620    5893 cpu_manager.go:414] "containerMap: removing container" podUID="21aac6b7-5d00-4753-9159-9c8c31172f95" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521628    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21aac6b7-5d00-4753-9159-9c8c31172f95" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521649    5893 cpu_manager.go:414] "containerMap: removing container" podUID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521656    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521671    5893 cpu_manager.go:414] "containerMap: removing container" podUID="926867a1-95ec-4376-98ab-c570de7f46bc" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521697    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926867a1-95ec-4376-98ab-c570de7f46bc" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521711    5893 cpu_manager.go:414] "containerMap: removing container" podUID="21aac6b7-5d00-4753-9159-9c8c31172f95" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521717    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21aac6b7-5d00-4753-9159-9c8c31172f95" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521736    5893 cpu_manager.go:414] "containerMap: removing container" podUID="926867a1-95ec-4376-98ab-c570de7f46bc" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521743    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="926867a1-95ec-4376-98ab-c570de7f46bc" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521751    5893 cpu_manager.go:414] "containerMap: removing container" podUID="9dc8baef-b3fd-410f-a85c-60006c08c3ad" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.521778    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dc8baef-b3fd-410f-a85c-60006c08c3ad" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.522039    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.522062    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.522092    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="21aac6b7-5d00-4753-9159-9c8c31172f95" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.522104    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6e949e-0b68-4c8e-bac6-3fccbf584cc4" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.522113    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="926867a1-95ec-4376-98ab-c570de7f46bc" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.522120    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="21aac6b7-5d00-4753-9159-9c8c31172f95" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.522132    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="926867a1-95ec-4376-98ab-c570de7f46bc" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.522142    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc8baef-b3fd-410f-a85c-60006c08c3ad" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.522177    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc8baef-b3fd-410f-a85c-60006c08c3ad" containerName="cleanup"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.522190    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cd3ca0-9231-47b7-8c2d-fb0b6ab88013" containerName="payload-processing"
Oct 30 13:15:45.522357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.522201    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dc8baef-b3fd-410f-a85c-60006c08c3ad" containerName="cleanup"
Oct 30 13:15:45.539464 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.539439    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"
Oct 30 13:15:45.567294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.551024    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"]
Oct 30 13:15:45.615196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.615145    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-6f9071\""
Oct 30 13:15:45.615426 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.615342    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-6f9071-dockercfg-llhxd\""
Oct 30 13:15:45.621947 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.621894    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww47z\" (UniqueName: \"kubernetes.io/projected/dc27afd0-973b-4e86-9ce1-adc6d294d7be-kube-api-access-ww47z\") pod \"payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz\" (UID: \"dc27afd0-973b-4e86-9ce1-adc6d294d7be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"
Oct 30 13:15:45.622884 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.622843    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc27afd0-973b-4e86-9ce1-adc6d294d7be-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz\" (UID: \"dc27afd0-973b-4e86-9ce1-adc6d294d7be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"
Oct 30 13:15:45.724798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.724765    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc27afd0-973b-4e86-9ce1-adc6d294d7be-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz\" (UID: \"dc27afd0-973b-4e86-9ce1-adc6d294d7be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"
Oct 30 13:15:45.724986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.724831    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww47z\" (UniqueName: \"kubernetes.io/projected/dc27afd0-973b-4e86-9ce1-adc6d294d7be-kube-api-access-ww47z\") pod \"payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz\" (UID: \"dc27afd0-973b-4e86-9ce1-adc6d294d7be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"
Oct 30 13:15:45.725523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.725501    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc27afd0-973b-4e86-9ce1-adc6d294d7be-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz\" (UID: \"dc27afd0-973b-4e86-9ce1-adc6d294d7be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"
Oct 30 13:15:45.777156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:45.777059    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww47z\" (UniqueName: \"kubernetes.io/projected/dc27afd0-973b-4e86-9ce1-adc6d294d7be-kube-api-access-ww47z\") pod \"payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz\" (UID: \"dc27afd0-973b-4e86-9ce1-adc6d294d7be\") " pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"
Oct 30 13:15:46.004249 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:46.004222    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"
Oct 30 13:15:46.386688 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:46.386625    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"]
Oct 30 13:15:46.390111 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:15:46.390062    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc27afd0_973b_4e86_9ce1_adc6d294d7be.slice/crio-cce0e660d5c2c64dcc537c4fdf1606237ae021a896c9b60077fd48968e941f07 WatchSource:0}: Error finding container cce0e660d5c2c64dcc537c4fdf1606237ae021a896c9b60077fd48968e941f07: Status 404 returned error can't find the container with id cce0e660d5c2c64dcc537c4fdf1606237ae021a896c9b60077fd48968e941f07
Oct 30 13:15:46.730350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:46.730298    5893 patch_prober.go:29] interesting pod/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:15:46.730743 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:46.730372    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn" podUID="3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:15:47.057821 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:47.056631    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz" event={"ID":"dc27afd0-973b-4e86-9ce1-adc6d294d7be","Type":"ContainerStarted","Data":"6a8a9507b0b5a4982d342fa59db603305d89065bd37279d6637b1893c5c2d1cd"}
Oct 30 13:15:47.057821 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:47.056698    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz" event={"ID":"dc27afd0-973b-4e86-9ce1-adc6d294d7be","Type":"ContainerStarted","Data":"cce0e660d5c2c64dcc537c4fdf1606237ae021a896c9b60077fd48968e941f07"}
Oct 30 13:15:47.057821 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:47.057709    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"
Oct 30 13:15:47.112381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:47.112323    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz" podStartSLOduration=2.112303917 podStartE2EDuration="2.112303917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:47.110475839 +0000 UTC m=+1788.693905532" watchObservedRunningTime="2026-10-30 13:15:47.112303917 +0000 UTC m=+1788.695733614"
Oct 30 13:15:48.722241 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:48.722184    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:15:48.819860 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.819835    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:15:48.902392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.902355    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-kube-api-access-xpxvw\" (UniqueName: \"kubernetes.io/projected/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-kube-api-access-xpxvw\") pod \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") "
Oct 30 13:15:48.902588 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.902438    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-data\") pod \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") "
Oct 30 13:15:48.902588 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.902460    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-workload-certs\") pod \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") "
Oct 30 13:15:48.902588 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.902516    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istiod-ca-cert\") pod \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") "
Oct 30 13:15:48.902588 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.902556    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-podinfo\") pod \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") "
Oct 30 13:15:48.902823 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.902605    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-envoy\") pod \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") "
Oct 30 13:15:48.902823 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.902626    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-workload-socket\") pod \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") "
Oct 30 13:15:48.902823 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.902651    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-token\" (UniqueName: \"kubernetes.io/projected/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-token\") pod \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") "
Oct 30 13:15:48.902823 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.902692    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-credential-socket\") pod \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\" (UID: \"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e\") "
Oct 30 13:15:48.903572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.903550    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istiod-ca-cert" pod "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" (UID: "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:48.904292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.904250    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-data" pod "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" (UID: "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:48.904397 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.904349    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-credential-socket" pod "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" (UID: "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:48.904461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.904420    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-workload-socket" pod "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" (UID: "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:48.904775 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.904734    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-workload-certs" pod "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" (UID: "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:48.909053 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.908961    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-podinfo" pod "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" (UID: "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:15:48.909053 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.909044    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-kube-api-access-xpxvw" pod "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" (UID: "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e"). InnerVolumeSpecName "kube-api-access-xpxvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:48.909053 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.909079    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-envoy" pod "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" (UID: "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:15:48.910563 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:48.910534    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-token" pod "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" (UID: "3b6b0cb8-104f-47f6-abda-9f34c5c1c12e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:49.004641 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.004608    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.004641 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.004642    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.004641 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.004657    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.004967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.004670    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.004967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.004686    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpxvw\" (UniqueName: \"kubernetes.io/projected/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-kube-api-access-xpxvw\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.004967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.004700    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.004967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.004713    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.004967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.004727    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.004967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.004738    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:15:49.079910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.079820    5893 generic.go:356] "Generic (PLEG): container finished" podID="3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" containerID="cb6e8e7285097ca05a81f20d35e3b89cbb2899d09edad40d05f6067b821a6ca5" exitCode=0
Oct 30 13:15:49.079910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.079872    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"
Oct 30 13:15:49.079910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.079900    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn" event={"ID":"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e","Type":"ContainerDied","Data":"cb6e8e7285097ca05a81f20d35e3b89cbb2899d09edad40d05f6067b821a6ca5"}
Oct 30 13:15:49.080179 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.079958    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn" event={"ID":"3b6b0cb8-104f-47f6-abda-9f34c5c1c12e","Type":"ContainerDied","Data":"ab7c0f9a34620044b93e17ce04d01324f9636b589f9d621aa2d5c19f98f53db7"}
Oct 30 13:15:49.080179 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.079982    5893 scope.go:122] "RemoveContainer" containerID="cb6e8e7285097ca05a81f20d35e3b89cbb2899d09edad40d05f6067b821a6ca5"
Oct 30 13:15:49.092866 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.092848    5893 scope.go:122] "RemoveContainer" containerID="cb6e8e7285097ca05a81f20d35e3b89cbb2899d09edad40d05f6067b821a6ca5"
Oct 30 13:15:49.093146 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:15:49.093125    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb6e8e7285097ca05a81f20d35e3b89cbb2899d09edad40d05f6067b821a6ca5\": container with ID starting with cb6e8e7285097ca05a81f20d35e3b89cbb2899d09edad40d05f6067b821a6ca5 not found: ID does not exist" containerID="cb6e8e7285097ca05a81f20d35e3b89cbb2899d09edad40d05f6067b821a6ca5"
Oct 30 13:15:49.093211 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.093158    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb6e8e7285097ca05a81f20d35e3b89cbb2899d09edad40d05f6067b821a6ca5"} err="failed to get container status \"cb6e8e7285097ca05a81f20d35e3b89cbb2899d09edad40d05f6067b821a6ca5\": rpc error: code = NotFound desc = could not find container \"cb6e8e7285097ca05a81f20d35e3b89cbb2899d09edad40d05f6067b821a6ca5\": container with ID starting with cb6e8e7285097ca05a81f20d35e3b89cbb2899d09edad40d05f6067b821a6ca5 not found: ID does not exist"
Oct 30 13:15:49.134298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.134225    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"]
Oct 30 13:15:49.139291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:49.138027    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-31e10b-openshift-default-5c5846b755-gxjnn"]
Oct 30 13:15:50.534090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:50.534049    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" path="/var/lib/kubelet/pods/3b6b0cb8-104f-47f6-abda-9f34c5c1c12e/volumes"
Oct 30 13:15:55.137812 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:55.137781    5893 generic.go:356] "Generic (PLEG): container finished" podID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerID="df7f17ae5f441d1ee4b66daffbcd1670bd2ce1084125d4ed86ef10bf1ab5d9e3" exitCode=0
Oct 30 13:15:55.138353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:55.137856    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h" event={"ID":"1ebcad4e-6749-4f92-9f85-6e72eb1a1954","Type":"ContainerDied","Data":"df7f17ae5f441d1ee4b66daffbcd1670bd2ce1084125d4ed86ef10bf1ab5d9e3"}
Oct 30 13:15:55.138435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:55.138407    5893 scope.go:122] "RemoveContainer" containerID="df7f17ae5f441d1ee4b66daffbcd1670bd2ce1084125d4ed86ef10bf1ab5d9e3"
Oct 30 13:15:55.140173 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:55.140143    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:15:55.140307 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:55.140229    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:15:55.140307 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:55.140300    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:15:56.149212 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:56.149172    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h" event={"ID":"1ebcad4e-6749-4f92-9f85-6e72eb1a1954","Type":"ContainerStarted","Data":"db327b06fe8e5b5dd39104f58b5ad3f28153e1e6040b64b07b3f403dbe2db1e1"}
Oct 30 13:15:57.020783 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:57.020741    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:15:57.021950 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:57.021924    5893 patch_prober.go:29] interesting pod/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:15:57.022073 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:57.022001    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h" podUID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 13:15:58.021785 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:58.021748    5893 patch_prober.go:29] interesting pod/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:15:58.022423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:58.021833    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h" podUID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 13:15:58.498556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:58.498522    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:15:59.021359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:59.021325    5893 patch_prober.go:29] interesting pod/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:15:59.021550 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:59.021388    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h" podUID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 13:15:59.084178 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:15:59.084088    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"
Oct 30 13:16:00.031942 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:00.031901    5893 patch_prober.go:29] interesting pod/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:16:00.032350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:00.032318    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h" podUID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 13:16:00.997368 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:00.997339    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:16:01.007662 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:01.007640    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:16:01.010616 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:01.010597    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:16:01.019945 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:01.019925    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:16:01.020956 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:01.020929    5893 patch_prober.go:29] interesting pod/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:16:01.021057 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:01.020975    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h" podUID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 13:16:02.021325 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:02.021292    5893 patch_prober.go:29] interesting pod/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 13:16:02.021718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:02.021352    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h" podUID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 13:16:03.026843 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:03.026810    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:16:03.028206 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:03.028187    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:16:18.648457 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:16:18.648421    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:16:32.550206 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:32.550173    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"]
Oct 30 13:16:32.551110 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:32.551079    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr" podUID="44d11615-8178-4efa-9468-ce4a6029e6db" containerName="maas-api" containerID="cri-o://ef02ebe5ee8a21251c343f7c9e29a9ac9bd1e8b9d490efe20a12e044447d3973" gracePeriod=30
Oct 30 13:16:32.888818 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:32.888759    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:16:32.941746 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:32.941168    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44d11615-8178-4efa-9468-ce4a6029e6db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44d11615-8178-4efa-9468-ce4a6029e6db-maas-api-tls\") pod \"44d11615-8178-4efa-9468-ce4a6029e6db\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") "
Oct 30 13:16:32.941746 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:32.941367    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d11615-8178-4efa-9468-ce4a6029e6db-kube-api-access-2cvgw\" (UniqueName: \"kubernetes.io/projected/44d11615-8178-4efa-9468-ce4a6029e6db-kube-api-access-2cvgw\") pod \"44d11615-8178-4efa-9468-ce4a6029e6db\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") "
Oct 30 13:16:32.941746 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:32.941567    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d11615-8178-4efa-9468-ce4a6029e6db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d11615-8178-4efa-9468-ce4a6029e6db-openshift-service-ca\") pod \"44d11615-8178-4efa-9468-ce4a6029e6db\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") "
Oct 30 13:16:32.941746 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:32.941606    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44d11615-8178-4efa-9468-ce4a6029e6db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44d11615-8178-4efa-9468-ce4a6029e6db-metrics-certs\") pod \"44d11615-8178-4efa-9468-ce4a6029e6db\" (UID: \"44d11615-8178-4efa-9468-ce4a6029e6db\") "
Oct 30 13:16:32.942985 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:32.942933    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d11615-8178-4efa-9468-ce4a6029e6db-openshift-service-ca" pod "44d11615-8178-4efa-9468-ce4a6029e6db" (UID: "44d11615-8178-4efa-9468-ce4a6029e6db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:32.950351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:32.950309    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d11615-8178-4efa-9468-ce4a6029e6db-kube-api-access-2cvgw" pod "44d11615-8178-4efa-9468-ce4a6029e6db" (UID: "44d11615-8178-4efa-9468-ce4a6029e6db"). InnerVolumeSpecName "kube-api-access-2cvgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:32.950503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:32.950352    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d11615-8178-4efa-9468-ce4a6029e6db-metrics-certs" pod "44d11615-8178-4efa-9468-ce4a6029e6db" (UID: "44d11615-8178-4efa-9468-ce4a6029e6db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:16:32.952173 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:32.952141    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d11615-8178-4efa-9468-ce4a6029e6db-maas-api-tls" pod "44d11615-8178-4efa-9468-ce4a6029e6db" (UID: "44d11615-8178-4efa-9468-ce4a6029e6db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:16:33.044179 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:33.044147    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44d11615-8178-4efa-9468-ce4a6029e6db-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:33.044179 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:33.044177    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44d11615-8178-4efa-9468-ce4a6029e6db-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:33.044179 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:33.044186    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44d11615-8178-4efa-9468-ce4a6029e6db-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:33.044486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:33.044194    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cvgw\" (UniqueName: \"kubernetes.io/projected/44d11615-8178-4efa-9468-ce4a6029e6db-kube-api-access-2cvgw\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:33.480264 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:33.476327    5893 generic.go:356] "Generic (PLEG): container finished" podID="44d11615-8178-4efa-9468-ce4a6029e6db" containerID="ef02ebe5ee8a21251c343f7c9e29a9ac9bd1e8b9d490efe20a12e044447d3973" exitCode=0
Oct 30 13:16:33.480264 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:33.476384    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"
Oct 30 13:16:33.480264 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:33.476403    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr" event={"ID":"44d11615-8178-4efa-9468-ce4a6029e6db","Type":"ContainerDied","Data":"ef02ebe5ee8a21251c343f7c9e29a9ac9bd1e8b9d490efe20a12e044447d3973"}
Oct 30 13:16:33.480264 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:33.476489    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr" event={"ID":"44d11615-8178-4efa-9468-ce4a6029e6db","Type":"ContainerDied","Data":"9fd53dee814a901891d5c90cbdba76d304ca860850a1d786451baa599e57a8a3"}
Oct 30 13:16:33.480264 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:33.476543    5893 scope.go:122] "RemoveContainer" containerID="ef02ebe5ee8a21251c343f7c9e29a9ac9bd1e8b9d490efe20a12e044447d3973"
Oct 30 13:16:33.494054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:33.494031    5893 scope.go:122] "RemoveContainer" containerID="ef02ebe5ee8a21251c343f7c9e29a9ac9bd1e8b9d490efe20a12e044447d3973"
Oct 30 13:16:33.494528 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:16:33.494493    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef02ebe5ee8a21251c343f7c9e29a9ac9bd1e8b9d490efe20a12e044447d3973\": container with ID starting with ef02ebe5ee8a21251c343f7c9e29a9ac9bd1e8b9d490efe20a12e044447d3973 not found: ID does not exist" containerID="ef02ebe5ee8a21251c343f7c9e29a9ac9bd1e8b9d490efe20a12e044447d3973"
Oct 30 13:16:33.494622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:33.494532    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef02ebe5ee8a21251c343f7c9e29a9ac9bd1e8b9d490efe20a12e044447d3973"} err="failed to get container status \"ef02ebe5ee8a21251c343f7c9e29a9ac9bd1e8b9d490efe20a12e044447d3973\": rpc error: code = NotFound desc = could not find container \"ef02ebe5ee8a21251c343f7c9e29a9ac9bd1e8b9d490efe20a12e044447d3973\": container with ID starting with ef02ebe5ee8a21251c343f7c9e29a9ac9bd1e8b9d490efe20a12e044447d3973 not found: ID does not exist"
Oct 30 13:16:33.538399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:33.538368    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"]
Oct 30 13:16:33.559933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:33.559900    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-abbc6c-664db4b855-dthxr"]
Oct 30 13:16:34.108031 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:34.107993    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"]
Oct 30 13:16:34.114703 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:34.114674    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-abbc6c-2p9xr"]
Oct 30 13:16:34.536754 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:34.536720    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d11615-8178-4efa-9468-ce4a6029e6db" path="/var/lib/kubelet/pods/44d11615-8178-4efa-9468-ce4a6029e6db/volumes"
Oct 30 13:16:34.537364 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:34.537338    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="783977cb-aa63-461f-9fb0-ee490af3989c" path="/var/lib/kubelet/pods/783977cb-aa63-461f-9fb0-ee490af3989c/volumes"
Oct 30 13:16:42.315084 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:42.315053    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"]
Oct 30 13:16:42.315704 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:42.315674    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerName="istio-proxy" containerID="cri-o://28bb9f875cd3d9dfcca5db15ccecd51caccadc70a15cca12b39b28ab78c4ba82" gracePeriod=30
Oct 30 13:16:42.653312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:42.653172    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"]
Oct 30 13:16:47.603723 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.603688    5893 generic.go:356] "Generic (PLEG): container finished" podID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerID="28bb9f875cd3d9dfcca5db15ccecd51caccadc70a15cca12b39b28ab78c4ba82" exitCode=0
Oct 30 13:16:47.604069 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.603758    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" event={"ID":"2ed508c1-ee5b-47e1-b335-7453b53a6f9d","Type":"ContainerDied","Data":"28bb9f875cd3d9dfcca5db15ccecd51caccadc70a15cca12b39b28ab78c4ba82"}
Oct 30 13:16:47.604069 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.603835    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw" event={"ID":"2ed508c1-ee5b-47e1-b335-7453b53a6f9d","Type":"ContainerDied","Data":"600739324be10af9e9252f1932dea815d25459f0897a2d42efe2a40f81fba523"}
Oct 30 13:16:47.604069 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.603861    5893 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="600739324be10af9e9252f1932dea815d25459f0897a2d42efe2a40f81fba523"
Oct 30 13:16:47.604069 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.603881    5893 scope.go:122] "RemoveContainer" containerID="0074433727b7e1099de818eb07ed8ab63cce1d10de7fdb2641bc9fc90a1b1b70"
Oct 30 13:16:47.620612 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.620587    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:16:47.654083 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.654039    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn" podUID="1425d7d5-e09a-4a70-98c9-58c50474a2da" containerName="payload-processing" containerID="cri-o://c95a4331cb5d47548f6de82b718572b26f8831c786f9b54a3f5ddf99fdbfaba6" gracePeriod=25
Oct 30 13:16:47.689906 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.689872    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-data\") pod \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") "
Oct 30 13:16:47.690077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.689930    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-credential-socket\") pod \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") "
Oct 30 13:16:47.690077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.689999    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-workload-socket\") pod \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") "
Oct 30 13:16:47.690245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.690191    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-data" pod "2ed508c1-ee5b-47e1-b335-7453b53a6f9d" (UID: "2ed508c1-ee5b-47e1-b335-7453b53a6f9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:47.690245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.690208    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-credential-socket" pod "2ed508c1-ee5b-47e1-b335-7453b53a6f9d" (UID: "2ed508c1-ee5b-47e1-b335-7453b53a6f9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:47.690407 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.690356    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-workload-socket" pod "2ed508c1-ee5b-47e1-b335-7453b53a6f9d" (UID: "2ed508c1-ee5b-47e1-b335-7453b53a6f9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:47.690467 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.690451    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-podinfo\") pod \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") "
Oct 30 13:16:47.690835 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.690809    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-workload-certs\") pod \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") "
Oct 30 13:16:47.690968 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.690922    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-workload-certs" pod "2ed508c1-ee5b-47e1-b335-7453b53a6f9d" (UID: "2ed508c1-ee5b-47e1-b335-7453b53a6f9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:47.691037 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.691001    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-token\" (UniqueName: \"kubernetes.io/projected/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-token\") pod \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") "
Oct 30 13:16:47.691145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.691126    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-envoy\") pod \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") "
Oct 30 13:16:47.691319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.691165    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-kube-api-access-lt8kj\" (UniqueName: \"kubernetes.io/projected/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-kube-api-access-lt8kj\") pod \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") "
Oct 30 13:16:47.691319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.691224    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istiod-ca-cert\") pod \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\" (UID: \"2ed508c1-ee5b-47e1-b335-7453b53a6f9d\") "
Oct 30 13:16:47.691840 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.691819    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istiod-ca-cert" pod "2ed508c1-ee5b-47e1-b335-7453b53a6f9d" (UID: "2ed508c1-ee5b-47e1-b335-7453b53a6f9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:47.693026 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.693001    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:47.693131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.693031    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:47.693131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.693047    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:47.693131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.693059    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:47.693131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.693073    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:47.695193 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.695151    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-token" pod "2ed508c1-ee5b-47e1-b335-7453b53a6f9d" (UID: "2ed508c1-ee5b-47e1-b335-7453b53a6f9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:47.695866 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.695831    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-envoy" pod "2ed508c1-ee5b-47e1-b335-7453b53a6f9d" (UID: "2ed508c1-ee5b-47e1-b335-7453b53a6f9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:16:47.695971 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.695867    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-podinfo" pod "2ed508c1-ee5b-47e1-b335-7453b53a6f9d" (UID: "2ed508c1-ee5b-47e1-b335-7453b53a6f9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:16:47.695971 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.695878    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-kube-api-access-lt8kj" pod "2ed508c1-ee5b-47e1-b335-7453b53a6f9d" (UID: "2ed508c1-ee5b-47e1-b335-7453b53a6f9d"). InnerVolumeSpecName "kube-api-access-lt8kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:47.795208 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.795124    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:47.795208 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.795155    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:47.795208 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.795169    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:47.795208 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.795183    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8kj\" (UniqueName: \"kubernetes.io/projected/2ed508c1-ee5b-47e1-b335-7453b53a6f9d-kube-api-access-lt8kj\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:47.921862 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.921838    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"
Oct 30 13:16:47.997686 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.997645    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1425d7d5-e09a-4a70-98c9-58c50474a2da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1425d7d5-e09a-4a70-98c9-58c50474a2da-plugins-config-volume\") pod \"1425d7d5-e09a-4a70-98c9-58c50474a2da\" (UID: \"1425d7d5-e09a-4a70-98c9-58c50474a2da\") "
Oct 30 13:16:47.998576 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.997894    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1425d7d5-e09a-4a70-98c9-58c50474a2da-kube-api-access-b22l6\" (UniqueName: \"kubernetes.io/projected/1425d7d5-e09a-4a70-98c9-58c50474a2da-kube-api-access-b22l6\") pod \"1425d7d5-e09a-4a70-98c9-58c50474a2da\" (UID: \"1425d7d5-e09a-4a70-98c9-58c50474a2da\") "
Oct 30 13:16:47.998576 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.998056    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1425d7d5-e09a-4a70-98c9-58c50474a2da-plugins-config-volume" pod "1425d7d5-e09a-4a70-98c9-58c50474a2da" (UID: "1425d7d5-e09a-4a70-98c9-58c50474a2da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:47.998576 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:47.998497    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1425d7d5-e09a-4a70-98c9-58c50474a2da-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:48.001127 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:48.001104    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1425d7d5-e09a-4a70-98c9-58c50474a2da-kube-api-access-b22l6" pod "1425d7d5-e09a-4a70-98c9-58c50474a2da" (UID: "1425d7d5-e09a-4a70-98c9-58c50474a2da"). InnerVolumeSpecName "kube-api-access-b22l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:48.100468 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:48.100430    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b22l6\" (UniqueName: \"kubernetes.io/projected/1425d7d5-e09a-4a70-98c9-58c50474a2da-kube-api-access-b22l6\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:48.612804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:48.612726    5893 generic.go:356] "Generic (PLEG): container finished" podID="1425d7d5-e09a-4a70-98c9-58c50474a2da" containerID="c95a4331cb5d47548f6de82b718572b26f8831c786f9b54a3f5ddf99fdbfaba6" exitCode=0
Oct 30 13:16:48.612804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:48.612774    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"
Oct 30 13:16:48.613363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:48.612807    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn" event={"ID":"1425d7d5-e09a-4a70-98c9-58c50474a2da","Type":"ContainerDied","Data":"c95a4331cb5d47548f6de82b718572b26f8831c786f9b54a3f5ddf99fdbfaba6"}
Oct 30 13:16:48.613363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:48.612856    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn" event={"ID":"1425d7d5-e09a-4a70-98c9-58c50474a2da","Type":"ContainerDied","Data":"1a8b406d6f4571dd505ac2df33a5e579f718521267e3aaabfc2d43f3ef266c7a"}
Oct 30 13:16:48.613363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:48.612883    5893 scope.go:122] "RemoveContainer" containerID="c95a4331cb5d47548f6de82b718572b26f8831c786f9b54a3f5ddf99fdbfaba6"
Oct 30 13:16:48.614672 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:48.614654    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"
Oct 30 13:16:48.623595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:48.623574    5893 scope.go:122] "RemoveContainer" containerID="c95a4331cb5d47548f6de82b718572b26f8831c786f9b54a3f5ddf99fdbfaba6"
Oct 30 13:16:48.623885 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:16:48.623863    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95a4331cb5d47548f6de82b718572b26f8831c786f9b54a3f5ddf99fdbfaba6\": container with ID starting with c95a4331cb5d47548f6de82b718572b26f8831c786f9b54a3f5ddf99fdbfaba6 not found: ID does not exist" containerID="c95a4331cb5d47548f6de82b718572b26f8831c786f9b54a3f5ddf99fdbfaba6"
Oct 30 13:16:48.623963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:48.623894    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95a4331cb5d47548f6de82b718572b26f8831c786f9b54a3f5ddf99fdbfaba6"} err="failed to get container status \"c95a4331cb5d47548f6de82b718572b26f8831c786f9b54a3f5ddf99fdbfaba6\": rpc error: code = NotFound desc = could not find container \"c95a4331cb5d47548f6de82b718572b26f8831c786f9b54a3f5ddf99fdbfaba6\": container with ID starting with c95a4331cb5d47548f6de82b718572b26f8831c786f9b54a3f5ddf99fdbfaba6 not found: ID does not exist"
Oct 30 13:16:48.649181 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:16:48.649151    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:16:48.694958 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:48.694925    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"]
Oct 30 13:16:48.702757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:48.702730    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-abbc6c-openshift-default-9f4f4d7ff-667fw"]
Oct 30 13:16:48.735965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:48.735940    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"]
Oct 30 13:16:48.742867 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:48.742844    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-14e8a4-7fb9b9d5d-mvqgn"]
Oct 30 13:16:50.536259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:50.536226    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1425d7d5-e09a-4a70-98c9-58c50474a2da" path="/var/lib/kubelet/pods/1425d7d5-e09a-4a70-98c9-58c50474a2da/volumes"
Oct 30 13:16:50.536790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:50.536774    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" path="/var/lib/kubelet/pods/2ed508c1-ee5b-47e1-b335-7453b53a6f9d/volumes"
Oct 30 13:16:53.199068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:53.198969    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"]
Oct 30 13:16:58.030460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.030418    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"]
Oct 30 13:16:58.031549 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.031528    5893 cpu_manager.go:414] "containerMap: removing container" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerName="istio-proxy"
Oct 30 13:16:58.031549 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.031548    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerName="istio-proxy"
Oct 30 13:16:58.031726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.031576    5893 cpu_manager.go:414] "containerMap: removing container" podUID="3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" containerName="istio-proxy"
Oct 30 13:16:58.031726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.031581    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" containerName="istio-proxy"
Oct 30 13:16:58.031726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.031590    5893 cpu_manager.go:414] "containerMap: removing container" podUID="1425d7d5-e09a-4a70-98c9-58c50474a2da" containerName="payload-processing"
Oct 30 13:16:58.031726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.031594    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1425d7d5-e09a-4a70-98c9-58c50474a2da" containerName="payload-processing"
Oct 30 13:16:58.031726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.031620    5893 cpu_manager.go:414] "containerMap: removing container" podUID="44d11615-8178-4efa-9468-ce4a6029e6db" containerName="maas-api"
Oct 30 13:16:58.031726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.031625    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d11615-8178-4efa-9468-ce4a6029e6db" containerName="maas-api"
Oct 30 13:16:58.032033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.031732    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d11615-8178-4efa-9468-ce4a6029e6db" containerName="maas-api"
Oct 30 13:16:58.032033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.031741    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="21aac6b7-5d00-4753-9159-9c8c31172f95" containerName="cleanup"
Oct 30 13:16:58.032033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.031750    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerName="istio-proxy"
Oct 30 13:16:58.032033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.031755    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="926867a1-95ec-4376-98ab-c570de7f46bc" containerName="cleanup"
Oct 30 13:16:58.032033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.031768    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6b0cb8-104f-47f6-abda-9f34c5c1c12e" containerName="istio-proxy"
Oct 30 13:16:58.032033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.031774    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="1425d7d5-e09a-4a70-98c9-58c50474a2da" containerName="payload-processing"
Oct 30 13:16:58.036539 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.036523    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.075128 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.075098    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"]
Oct 30 13:16:58.124643 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.124604    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cd2d2e-openshift-default-dockercfg-fc475\""
Oct 30 13:16:58.168608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.168573    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k496z\" (UniqueName: \"kubernetes.io/projected/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-kube-api-access-k496z\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.168942 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.168905    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-workload-certs\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.169132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.169113    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istiod-ca-cert\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.169879 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.169748    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-workload-socket\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.170224 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.170206    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-podinfo\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.171567 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.171547    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-credential-socket\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.172004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.171985    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-data\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.172312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.172295    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-token\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.172617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.172572    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-envoy\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.199890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.199846    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk" podUID="7e10009d-973a-440c-aa8f-89aa1ce91755" containerName="payload-processing" containerID="cri-o://4a3ea66edacdb048e09e77eef40ea7778d9789625c289b5e13704ad2dd49bba3" gracePeriod=25
Oct 30 13:16:58.274406 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.274366    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-token\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.274595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.274439    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-envoy\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.274595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.274529    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k496z\" (UniqueName: \"kubernetes.io/projected/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-kube-api-access-k496z\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.274595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.274565    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-workload-certs\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.274772 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.274598    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istiod-ca-cert\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.274772 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.274694    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-workload-socket\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.274881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.274784    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-podinfo\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.274881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.274860    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-credential-socket\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.274995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.274902    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-data\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.275048 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.275011    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-workload-certs\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.275098 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.275054    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-workload-socket\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.275253 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.275234    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-data\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.275346 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.275233    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-credential-socket\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.275474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.275455    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istiod-ca-cert\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.277350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.277326    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-envoy\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.277568 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.277549    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-podinfo\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.303072 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.303012    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k496z\" (UniqueName: \"kubernetes.io/projected/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-kube-api-access-k496z\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.304046 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.304029    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-token\") pod \"e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") " pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.355300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.353365    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:16:58.544387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.544354    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"]
Oct 30 13:16:58.544720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.544688    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz" podUID="57facfc2-95f0-41d0-ab18-1b60dab9bc9d" containerName="maas-api" containerID="cri-o://5ab735c333881903ed196c7f9137569e2d4be3f79b556f6211aa2de2969ee76a" gracePeriod=30
Oct 30 13:16:58.670460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.670398    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"
Oct 30 13:16:58.726983 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.726949    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"]
Oct 30 13:16:58.729292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.727415    5893 generic.go:356] "Generic (PLEG): container finished" podID="7e10009d-973a-440c-aa8f-89aa1ce91755" containerID="4a3ea66edacdb048e09e77eef40ea7778d9789625c289b5e13704ad2dd49bba3" exitCode=0
Oct 30 13:16:58.729292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.727482    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"
Oct 30 13:16:58.729292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.727687    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk" event={"ID":"7e10009d-973a-440c-aa8f-89aa1ce91755","Type":"ContainerDied","Data":"4a3ea66edacdb048e09e77eef40ea7778d9789625c289b5e13704ad2dd49bba3"}
Oct 30 13:16:58.729292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.727730    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk" event={"ID":"7e10009d-973a-440c-aa8f-89aa1ce91755","Type":"ContainerDied","Data":"e70e6642f8d0d10fb4abd070d2b2b464e292b4c9ad9b445ad268ef5c632ec5c2"}
Oct 30 13:16:58.729292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.727759    5893 scope.go:122] "RemoveContainer" containerID="4a3ea66edacdb048e09e77eef40ea7778d9789625c289b5e13704ad2dd49bba3"
Oct 30 13:16:58.730628 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.730602    5893 generic.go:356] "Generic (PLEG): container finished" podID="57facfc2-95f0-41d0-ab18-1b60dab9bc9d" containerID="5ab735c333881903ed196c7f9137569e2d4be3f79b556f6211aa2de2969ee76a" exitCode=0
Oct 30 13:16:58.730845 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.730828    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz" event={"ID":"57facfc2-95f0-41d0-ab18-1b60dab9bc9d","Type":"ContainerDied","Data":"5ab735c333881903ed196c7f9137569e2d4be3f79b556f6211aa2de2969ee76a"}
Oct 30 13:16:58.797062 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.797020    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e10009d-973a-440c-aa8f-89aa1ce91755-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e10009d-973a-440c-aa8f-89aa1ce91755-plugins-config-volume\") pod \"7e10009d-973a-440c-aa8f-89aa1ce91755\" (UID: \"7e10009d-973a-440c-aa8f-89aa1ce91755\") "
Oct 30 13:16:58.797291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.797116    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e10009d-973a-440c-aa8f-89aa1ce91755-kube-api-access-7nqbt\" (UniqueName: \"kubernetes.io/projected/7e10009d-973a-440c-aa8f-89aa1ce91755-kube-api-access-7nqbt\") pod \"7e10009d-973a-440c-aa8f-89aa1ce91755\" (UID: \"7e10009d-973a-440c-aa8f-89aa1ce91755\") "
Oct 30 13:16:58.818863 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.808473    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e10009d-973a-440c-aa8f-89aa1ce91755-plugins-config-volume" pod "7e10009d-973a-440c-aa8f-89aa1ce91755" (UID: "7e10009d-973a-440c-aa8f-89aa1ce91755"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:58.818863 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.808784    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e10009d-973a-440c-aa8f-89aa1ce91755-kube-api-access-7nqbt" pod "7e10009d-973a-440c-aa8f-89aa1ce91755" (UID: "7e10009d-973a-440c-aa8f-89aa1ce91755"). InnerVolumeSpecName "kube-api-access-7nqbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:58.825786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.823349    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:16:58.825786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.823452    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:16:58.825786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.823497    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:16:58.832495 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.830860    5893 scope.go:122] "RemoveContainer" containerID="4a3ea66edacdb048e09e77eef40ea7778d9789625c289b5e13704ad2dd49bba3"
Oct 30 13:16:58.834157 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:16:58.833932    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3ea66edacdb048e09e77eef40ea7778d9789625c289b5e13704ad2dd49bba3\": container with ID starting with 4a3ea66edacdb048e09e77eef40ea7778d9789625c289b5e13704ad2dd49bba3 not found: ID does not exist" containerID="4a3ea66edacdb048e09e77eef40ea7778d9789625c289b5e13704ad2dd49bba3"
Oct 30 13:16:58.834157 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.833997    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3ea66edacdb048e09e77eef40ea7778d9789625c289b5e13704ad2dd49bba3"} err="failed to get container status \"4a3ea66edacdb048e09e77eef40ea7778d9789625c289b5e13704ad2dd49bba3\": rpc error: code = NotFound desc = could not find container \"4a3ea66edacdb048e09e77eef40ea7778d9789625c289b5e13704ad2dd49bba3\": container with ID starting with 4a3ea66edacdb048e09e77eef40ea7778d9789625c289b5e13704ad2dd49bba3 not found: ID does not exist"
Oct 30 13:16:58.900985 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.900958    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nqbt\" (UniqueName: \"kubernetes.io/projected/7e10009d-973a-440c-aa8f-89aa1ce91755-kube-api-access-7nqbt\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:58.901197 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.901179    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e10009d-973a-440c-aa8f-89aa1ce91755-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:58.990898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:58.990871    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:16:59.106445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.104027    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-kube-api-access-6rt4t\" (UniqueName: \"kubernetes.io/projected/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-kube-api-access-6rt4t\") pod \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") "
Oct 30 13:16:59.106445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.104155    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-maas-api-tls\") pod \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") "
Oct 30 13:16:59.106445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.104189    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-openshift-service-ca\") pod \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") "
Oct 30 13:16:59.106445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.104408    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-metrics-certs\") pod \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\" (UID: \"57facfc2-95f0-41d0-ab18-1b60dab9bc9d\") "
Oct 30 13:16:59.107305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.107256    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-metrics-certs" pod "57facfc2-95f0-41d0-ab18-1b60dab9bc9d" (UID: "57facfc2-95f0-41d0-ab18-1b60dab9bc9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:16:59.109450 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.109423    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-kube-api-access-6rt4t" pod "57facfc2-95f0-41d0-ab18-1b60dab9bc9d" (UID: "57facfc2-95f0-41d0-ab18-1b60dab9bc9d"). InnerVolumeSpecName "kube-api-access-6rt4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:59.111361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.111334    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-maas-api-tls" pod "57facfc2-95f0-41d0-ab18-1b60dab9bc9d" (UID: "57facfc2-95f0-41d0-ab18-1b60dab9bc9d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:16:59.111637 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.111618    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-openshift-service-ca" pod "57facfc2-95f0-41d0-ab18-1b60dab9bc9d" (UID: "57facfc2-95f0-41d0-ab18-1b60dab9bc9d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:59.133202 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.133170    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"]
Oct 30 13:16:59.137604 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.137574    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-11cfa6-689984f4b8-dmtfk"]
Oct 30 13:16:59.206180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.206145    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:59.206180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.206179    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rt4t\" (UniqueName: \"kubernetes.io/projected/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-kube-api-access-6rt4t\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:59.206424 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.206193    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:59.206424 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.206207    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57facfc2-95f0-41d0-ab18-1b60dab9bc9d-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:16:59.742364 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.742170    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb" event={"ID":"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611","Type":"ContainerStarted","Data":"c36824bd757c6bdd5c0783ba26bc38c0f734b8a1017fac5f9053699202edd58c"}
Oct 30 13:16:59.742364 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.742257    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb" event={"ID":"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611","Type":"ContainerStarted","Data":"98070b0daaf6beadeee5db5fc9997446259534c0858c90f93ceb6b3b0d460610"}
Oct 30 13:16:59.744317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.744249    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz" event={"ID":"57facfc2-95f0-41d0-ab18-1b60dab9bc9d","Type":"ContainerDied","Data":"d499a888fda72a636c520d23e8a0e86b71955927dd83f3cdbfd0567682913943"}
Oct 30 13:16:59.744317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.744288    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"
Oct 30 13:16:59.744539 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.744341    5893 scope.go:122] "RemoveContainer" containerID="5ab735c333881903ed196c7f9137569e2d4be3f79b556f6211aa2de2969ee76a"
Oct 30 13:16:59.806501 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.806448    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb" podStartSLOduration=2.8064336389999998 podStartE2EDuration="2.806433639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:59.804079961 +0000 UTC m=+1861.387509653" watchObservedRunningTime="2026-10-30 13:16:59.806433639 +0000 UTC m=+1861.389863332"
Oct 30 13:16:59.851967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.851933    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"]
Oct 30 13:16:59.859581 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:16:59.857048    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-716b4c-748fc76498-r9rsz"]
Oct 30 13:17:00.356077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:00.356041    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:17:00.363229 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:00.363205    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:17:00.533906 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:00.533877    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57facfc2-95f0-41d0-ab18-1b60dab9bc9d" path="/var/lib/kubelet/pods/57facfc2-95f0-41d0-ab18-1b60dab9bc9d/volumes"
Oct 30 13:17:00.534345 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:00.534331    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e10009d-973a-440c-aa8f-89aa1ce91755" path="/var/lib/kubelet/pods/7e10009d-973a-440c-aa8f-89aa1ce91755/volumes"
Oct 30 13:17:00.753235 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:00.753210    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:17:00.754318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:00.754261    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:17:04.409754 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.409718    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"]
Oct 30 13:17:04.410500 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.410484    5893 cpu_manager.go:414] "containerMap: removing container" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerName="istio-proxy"
Oct 30 13:17:04.410571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.410502    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerName="istio-proxy"
Oct 30 13:17:04.410571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.410511    5893 cpu_manager.go:414] "containerMap: removing container" podUID="57facfc2-95f0-41d0-ab18-1b60dab9bc9d" containerName="maas-api"
Oct 30 13:17:04.410571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.410517    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57facfc2-95f0-41d0-ab18-1b60dab9bc9d" containerName="maas-api"
Oct 30 13:17:04.410571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.410547    5893 cpu_manager.go:414] "containerMap: removing container" podUID="7e10009d-973a-440c-aa8f-89aa1ce91755" containerName="payload-processing"
Oct 30 13:17:04.410571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.410551    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e10009d-973a-440c-aa8f-89aa1ce91755" containerName="payload-processing"
Oct 30 13:17:04.410726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.410650    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e10009d-973a-440c-aa8f-89aa1ce91755" containerName="payload-processing"
Oct 30 13:17:04.410726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.410662    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed508c1-ee5b-47e1-b335-7453b53a6f9d" containerName="istio-proxy"
Oct 30 13:17:04.410726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.410673    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="57facfc2-95f0-41d0-ab18-1b60dab9bc9d" containerName="maas-api"
Oct 30 13:17:04.415848 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.415832    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"
Oct 30 13:17:04.436880 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.436857    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"]
Oct 30 13:17:04.446581 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.445298    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cd2d2e-dockercfg-v5spv\""
Oct 30 13:17:04.446581 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.446225    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cd2d2e\""
Oct 30 13:17:04.603746 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.603710    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59zr\" (UniqueName: \"kubernetes.io/projected/77464420-74b0-4fb5-a533-3ad3a79ff914-kube-api-access-f59zr\") pod \"payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k\" (UID: \"77464420-74b0-4fb5-a533-3ad3a79ff914\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"
Oct 30 13:17:04.604246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.604222    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77464420-74b0-4fb5-a533-3ad3a79ff914-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k\" (UID: \"77464420-74b0-4fb5-a533-3ad3a79ff914\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"
Oct 30 13:17:04.653763 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.653729    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"]
Oct 30 13:17:04.676384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.672760    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"]
Oct 30 13:17:04.676645 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.674170    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"
Oct 30 13:17:04.706203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.706171    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59zr\" (UniqueName: \"kubernetes.io/projected/77464420-74b0-4fb5-a533-3ad3a79ff914-kube-api-access-f59zr\") pod \"payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k\" (UID: \"77464420-74b0-4fb5-a533-3ad3a79ff914\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"
Oct 30 13:17:04.706571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.706545    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77464420-74b0-4fb5-a533-3ad3a79ff914-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k\" (UID: \"77464420-74b0-4fb5-a533-3ad3a79ff914\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"
Oct 30 13:17:04.707582 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.707560    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77464420-74b0-4fb5-a533-3ad3a79ff914-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k\" (UID: \"77464420-74b0-4fb5-a533-3ad3a79ff914\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"
Oct 30 13:17:04.731622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.731596    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59zr\" (UniqueName: \"kubernetes.io/projected/77464420-74b0-4fb5-a533-3ad3a79ff914-kube-api-access-f59zr\") pod \"payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k\" (UID: \"77464420-74b0-4fb5-a533-3ad3a79ff914\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"
Oct 30 13:17:04.807912 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.807874    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cae94263-d227-4d32-9acd-bb4615ce48d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg\" (UID: \"cae94263-d227-4d32-9acd-bb4615ce48d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"
Oct 30 13:17:04.808084 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.808035    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfr7r\" (UniqueName: \"kubernetes.io/projected/cae94263-d227-4d32-9acd-bb4615ce48d3-kube-api-access-cfr7r\") pod \"payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg\" (UID: \"cae94263-d227-4d32-9acd-bb4615ce48d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"
Oct 30 13:17:04.909415 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.909383    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cae94263-d227-4d32-9acd-bb4615ce48d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg\" (UID: \"cae94263-d227-4d32-9acd-bb4615ce48d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"
Oct 30 13:17:04.909608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.909524    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfr7r\" (UniqueName: \"kubernetes.io/projected/cae94263-d227-4d32-9acd-bb4615ce48d3-kube-api-access-cfr7r\") pod \"payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg\" (UID: \"cae94263-d227-4d32-9acd-bb4615ce48d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"
Oct 30 13:17:04.910151 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.910129    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cae94263-d227-4d32-9acd-bb4615ce48d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg\" (UID: \"cae94263-d227-4d32-9acd-bb4615ce48d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"
Oct 30 13:17:04.963503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.963473    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfr7r\" (UniqueName: \"kubernetes.io/projected/cae94263-d227-4d32-9acd-bb4615ce48d3-kube-api-access-cfr7r\") pod \"payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg\" (UID: \"cae94263-d227-4d32-9acd-bb4615ce48d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"
Oct 30 13:17:04.996432 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:04.996403    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"
Oct 30 13:17:05.030882 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:05.030852    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"
Oct 30 13:17:05.313831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:05.313807    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"]
Oct 30 13:17:05.315586 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:17:05.315562    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae94263_d227_4d32_9acd_bb4615ce48d3.slice/crio-4d7564f670e1f2aef862fd2d5397e7021be68d7379310db50ccd2c0493383bc1 WatchSource:0}: Error finding container 4d7564f670e1f2aef862fd2d5397e7021be68d7379310db50ccd2c0493383bc1: Status 404 returned error can't find the container with id 4d7564f670e1f2aef862fd2d5397e7021be68d7379310db50ccd2c0493383bc1
Oct 30 13:17:05.358191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:05.358168    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"]
Oct 30 13:17:05.359680 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:17:05.359647    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77464420_74b0_4fb5_a533_3ad3a79ff914.slice/crio-37460eb47cd81c559a15736d511a53e7b3aef59f3592b452bf5f2564981ecfe9 WatchSource:0}: Error finding container 37460eb47cd81c559a15736d511a53e7b3aef59f3592b452bf5f2564981ecfe9: Status 404 returned error can't find the container with id 37460eb47cd81c559a15736d511a53e7b3aef59f3592b452bf5f2564981ecfe9
Oct 30 13:17:05.796160 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:05.796122    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg" event={"ID":"cae94263-d227-4d32-9acd-bb4615ce48d3","Type":"ContainerStarted","Data":"c34761c162709a163f997eda58b05c8c63fb52ff603f3e09830a7a71299e9445"}
Oct 30 13:17:05.796704 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:05.796678    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg" event={"ID":"cae94263-d227-4d32-9acd-bb4615ce48d3","Type":"ContainerStarted","Data":"4d7564f670e1f2aef862fd2d5397e7021be68d7379310db50ccd2c0493383bc1"}
Oct 30 13:17:05.796811 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:05.796731    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"
Oct 30 13:17:05.798108 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:05.798076    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k" event={"ID":"77464420-74b0-4fb5-a533-3ad3a79ff914","Type":"ContainerStarted","Data":"d53be3e3f9b4a9f3705a0b02bc584c81e492cf18284c29c8c438029a8c3c9b21"}
Oct 30 13:17:05.798233 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:05.798118    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k" event={"ID":"77464420-74b0-4fb5-a533-3ad3a79ff914","Type":"ContainerStarted","Data":"37460eb47cd81c559a15736d511a53e7b3aef59f3592b452bf5f2564981ecfe9"}
Oct 30 13:17:05.798233 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:05.798153    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"
Oct 30 13:17:05.840708 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:05.840646    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg" podStartSLOduration=1.840632109 podStartE2EDuration="1.840632109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:05.838202134 +0000 UTC m=+1867.421631827" watchObservedRunningTime="2026-10-30 13:17:05.840632109 +0000 UTC m=+1867.424061802"
Oct 30 13:17:05.880684 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:05.880623    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k" podStartSLOduration=1.880606139 podStartE2EDuration="1.880606139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:05.880059315 +0000 UTC m=+1867.463489008" watchObservedRunningTime="2026-10-30 13:17:05.880606139 +0000 UTC m=+1867.464035833"
Oct 30 13:17:06.896680 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:06.896631    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"]
Oct 30 13:17:06.897183 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:06.896892    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerName="istio-proxy" containerID="cri-o://82304be2cf824b2c0d13453744b2349d753d0ac928660a9e7563007bc631ca41" gracePeriod=30
Oct 30 13:17:08.167364 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.161798    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"]
Oct 30 13:17:08.168196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.168168    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.189721 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.189694    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-618c72-openshift-default-dockercfg-f98p9\""
Oct 30 13:17:08.206876 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.206846    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"]
Oct 30 13:17:08.261938 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.261903    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b6b218-4523-481c-8f22-b05a6e80af0a-istiod-ca-cert\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.262140 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.262111    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-token\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.262196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.262175    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-podinfo\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.262334 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.262315    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-data\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.262403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.262385    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-workload-certs\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.262437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.262418    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-credential-socket\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.262481 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.262442    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-envoy\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.262567 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.262547    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p47f2\" (UniqueName: \"kubernetes.io/projected/68b6b218-4523-481c-8f22-b05a6e80af0a-kube-api-access-p47f2\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.262611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.262593    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-workload-socket\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.368772 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.368728    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-workload-certs\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.388297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.377439    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-credential-socket\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.388572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.388548    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-envoy\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.389257 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.389235    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p47f2\" (UniqueName: \"kubernetes.io/projected/68b6b218-4523-481c-8f22-b05a6e80af0a-kube-api-access-p47f2\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.389471 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.389454    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-workload-socket\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.391337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.389663    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b6b218-4523-481c-8f22-b05a6e80af0a-istiod-ca-cert\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.391990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.391964    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-token\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.392437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.392417    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-podinfo\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.418618 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.418529    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-data\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.419375 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.370563    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-workload-certs\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.419532 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.390622    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-workload-socket\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.419637 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.385708    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-credential-socket\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.419733 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.391221    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b6b218-4523-481c-8f22-b05a6e80af0a-istiod-ca-cert\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.419831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.419137    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-data\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.427064 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.427002    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-envoy\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.430559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.430485    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-podinfo\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.452073 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.452042    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-token\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.457607 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.457543    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p47f2\" (UniqueName: \"kubernetes.io/projected/68b6b218-4523-481c-8f22-b05a6e80af0a-kube-api-access-p47f2\") pod \"e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") " pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.485354 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.485030    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:08.832234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.832209    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"]
Oct 30 13:17:08.833870 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:17:08.833845    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b6b218_4523_481c_8f22_b05a6e80af0a.slice/crio-bdfc9414b60fc579129089a22f1abcc89149653198eb663075d1ce59372df772 WatchSource:0}: Error finding container bdfc9414b60fc579129089a22f1abcc89149653198eb663075d1ce59372df772: Status 404 returned error can't find the container with id bdfc9414b60fc579129089a22f1abcc89149653198eb663075d1ce59372df772
Oct 30 13:17:08.836583 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.836552    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:17:08.836675 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.836613    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:17:08.836675 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:08.836640    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:17:09.843472 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:09.843432    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" event={"ID":"68b6b218-4523-481c-8f22-b05a6e80af0a","Type":"ContainerStarted","Data":"18234e525c3631c355702e50f4a6dc0e387feca51f1d515e4c4bba2bcdebc11f"}
Oct 30 13:17:09.843909 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:09.843498    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" event={"ID":"68b6b218-4523-481c-8f22-b05a6e80af0a","Type":"ContainerStarted","Data":"bdfc9414b60fc579129089a22f1abcc89149653198eb663075d1ce59372df772"}
Oct 30 13:17:09.909958 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:09.909905    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" podStartSLOduration=1.9098903360000001 podStartE2EDuration="1.909890336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:09.907196176 +0000 UTC m=+1871.490625868" watchObservedRunningTime="2026-10-30 13:17:09.909890336 +0000 UTC m=+1871.493320098"
Oct 30 13:17:10.486355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:10.486320    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:10.493267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:10.493237    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:10.851149 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:10.851052    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:10.852328 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:10.852309    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:17:11.440528 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:11.440490    5893 patch_prober.go:29] interesting pod/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:17:11.440725 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:11.440553    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:17:12.191036 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.191011    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:17:12.296406 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.296369    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-workload-socket\") pod \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") "
Oct 30 13:17:12.296614 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.296432    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-credential-socket\") pod \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") "
Oct 30 13:17:12.296614 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.296574    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-kube-api-access-lhnzf\" (UniqueName: \"kubernetes.io/projected/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-kube-api-access-lhnzf\") pod \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") "
Oct 30 13:17:12.296737 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.296621    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-data\") pod \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") "
Oct 30 13:17:12.296737 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.296696    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-workload-socket" pod "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" (UID: "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:12.296858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.296747    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-envoy\") pod \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") "
Oct 30 13:17:12.296858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.296832    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-credential-socket" pod "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" (UID: "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:12.296858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.296831    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-podinfo\") pod \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") "
Oct 30 13:17:12.297171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.296876    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istiod-ca-cert\") pod \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") "
Oct 30 13:17:12.297171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.296945    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-token\" (UniqueName: \"kubernetes.io/projected/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-token\") pod \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") "
Oct 30 13:17:12.297171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.296975    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-workload-certs\") pod \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\" (UID: \"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a\") "
Oct 30 13:17:12.297171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.297007    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-data" pod "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" (UID: "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:12.297669 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.297643    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:17:12.297669 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.297669    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:17:12.298286 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.297683    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:17:12.298286 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.297902    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-workload-certs" pod "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" (UID: "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:12.298286 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.297989    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istiod-ca-cert" pod "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" (UID: "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:12.300313 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.299827    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-kube-api-access-lhnzf" pod "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" (UID: "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a"). InnerVolumeSpecName "kube-api-access-lhnzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:12.300313 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.299957    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-podinfo" pod "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" (UID: "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:17:12.300313 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.300048    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-envoy" pod "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" (UID: "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:12.301730 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.301704    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-token" pod "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" (UID: "eec70a1d-58a8-4db9-ac0e-a7a08cefd63a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:12.399068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.399034    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:17:12.399068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.399064    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:17:12.399068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.399074    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:17:12.399343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.399083    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:17:12.399343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.399091    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:17:12.399343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.399099    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhnzf\" (UniqueName: \"kubernetes.io/projected/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a-kube-api-access-lhnzf\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:17:12.871128 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.871032    5893 generic.go:356] "Generic (PLEG): container finished" podID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerID="82304be2cf824b2c0d13453744b2349d753d0ac928660a9e7563007bc631ca41" exitCode=0
Oct 30 13:17:12.871313 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.871126    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"
Oct 30 13:17:12.871313 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.871131    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" event={"ID":"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a","Type":"ContainerDied","Data":"82304be2cf824b2c0d13453744b2349d753d0ac928660a9e7563007bc631ca41"}
Oct 30 13:17:12.871313 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.871206    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk" event={"ID":"eec70a1d-58a8-4db9-ac0e-a7a08cefd63a","Type":"ContainerDied","Data":"1a7bbc97be2967abe4335a16a4dad262c6c844323205f54bfdeb051a5826daa1"}
Oct 30 13:17:12.871313 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.871236    5893 scope.go:122] "RemoveContainer" containerID="82304be2cf824b2c0d13453744b2349d753d0ac928660a9e7563007bc631ca41"
Oct 30 13:17:12.882418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.882398    5893 scope.go:122] "RemoveContainer" containerID="23e2df560a06d0096ca32c0be19a7dd026b7b03e11c01a4967e34d44c277ac02"
Oct 30 13:17:12.892769 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.892741    5893 scope.go:122] "RemoveContainer" containerID="82304be2cf824b2c0d13453744b2349d753d0ac928660a9e7563007bc631ca41"
Oct 30 13:17:12.893040 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:17:12.893018    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82304be2cf824b2c0d13453744b2349d753d0ac928660a9e7563007bc631ca41\": container with ID starting with 82304be2cf824b2c0d13453744b2349d753d0ac928660a9e7563007bc631ca41 not found: ID does not exist" containerID="82304be2cf824b2c0d13453744b2349d753d0ac928660a9e7563007bc631ca41"
Oct 30 13:17:12.893107 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.893050    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82304be2cf824b2c0d13453744b2349d753d0ac928660a9e7563007bc631ca41"} err="failed to get container status \"82304be2cf824b2c0d13453744b2349d753d0ac928660a9e7563007bc631ca41\": rpc error: code = NotFound desc = could not find container \"82304be2cf824b2c0d13453744b2349d753d0ac928660a9e7563007bc631ca41\": container with ID starting with 82304be2cf824b2c0d13453744b2349d753d0ac928660a9e7563007bc631ca41 not found: ID does not exist"
Oct 30 13:17:12.893107 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.893067    5893 scope.go:122] "RemoveContainer" containerID="23e2df560a06d0096ca32c0be19a7dd026b7b03e11c01a4967e34d44c277ac02"
Oct 30 13:17:12.893350 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:17:12.893334    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e2df560a06d0096ca32c0be19a7dd026b7b03e11c01a4967e34d44c277ac02\": container with ID starting with 23e2df560a06d0096ca32c0be19a7dd026b7b03e11c01a4967e34d44c277ac02 not found: ID does not exist" containerID="23e2df560a06d0096ca32c0be19a7dd026b7b03e11c01a4967e34d44c277ac02"
Oct 30 13:17:12.893404 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.893354    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e2df560a06d0096ca32c0be19a7dd026b7b03e11c01a4967e34d44c277ac02"} err="failed to get container status \"23e2df560a06d0096ca32c0be19a7dd026b7b03e11c01a4967e34d44c277ac02\": rpc error: code = NotFound desc = could not find container \"23e2df560a06d0096ca32c0be19a7dd026b7b03e11c01a4967e34d44c277ac02\": container with ID starting with 23e2df560a06d0096ca32c0be19a7dd026b7b03e11c01a4967e34d44c277ac02 not found: ID does not exist"
Oct 30 13:17:12.938091 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.938057    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"]
Oct 30 13:17:12.949161 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:12.949137    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-716b4c-openshift-default-55bb8ffb77-xr5gk"]
Oct 30 13:17:14.533157 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.533129    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" path="/var/lib/kubelet/pods/eec70a1d-58a8-4db9-ac0e-a7a08cefd63a/volumes"
Oct 30 13:17:14.582337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.582299    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"]
Oct 30 13:17:14.583035 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.583019    5893 cpu_manager.go:414] "containerMap: removing container" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerName="istio-proxy"
Oct 30 13:17:14.583035 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.583035    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerName="istio-proxy"
Oct 30 13:17:14.583194 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.583060    5893 cpu_manager.go:414] "containerMap: removing container" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerName="istio-proxy"
Oct 30 13:17:14.583194 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.583065    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerName="istio-proxy"
Oct 30 13:17:14.583194 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.583169    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerName="istio-proxy"
Oct 30 13:17:14.583194 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.583182    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec70a1d-58a8-4db9-ac0e-a7a08cefd63a" containerName="istio-proxy"
Oct 30 13:17:14.588560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.588538    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"
Oct 30 13:17:14.610493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.610466    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-618c72\""
Oct 30 13:17:14.622692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.621948    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-618c72-dockercfg-6zc54\""
Oct 30 13:17:14.625642 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.625617    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"]
Oct 30 13:17:14.684480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.674490    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"]
Oct 30 13:17:14.750777 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.745212    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3e89b62-a248-46fc-8e83-fc2d4fa66a02-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb\" (UID: \"a3e89b62-a248-46fc-8e83-fc2d4fa66a02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"
Oct 30 13:17:14.750777 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.745426    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgbs5\" (UniqueName: \"kubernetes.io/projected/a3e89b62-a248-46fc-8e83-fc2d4fa66a02-kube-api-access-rgbs5\") pod \"payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb\" (UID: \"a3e89b62-a248-46fc-8e83-fc2d4fa66a02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"
Oct 30 13:17:14.757580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.757544    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"
Oct 30 13:17:14.811539 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.798164    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"]
Oct 30 13:17:14.855439 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.855396    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgbs5\" (UniqueName: \"kubernetes.io/projected/a3e89b62-a248-46fc-8e83-fc2d4fa66a02-kube-api-access-rgbs5\") pod \"payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb\" (UID: \"a3e89b62-a248-46fc-8e83-fc2d4fa66a02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"
Oct 30 13:17:14.855661 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.855480    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a814d069-870d-4ea0-945f-2ddcd1d517eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq\" (UID: \"a814d069-870d-4ea0-945f-2ddcd1d517eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"
Oct 30 13:17:14.855661 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.855620    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj48b\" (UniqueName: \"kubernetes.io/projected/a814d069-870d-4ea0-945f-2ddcd1d517eb-kube-api-access-mj48b\") pod \"payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq\" (UID: \"a814d069-870d-4ea0-945f-2ddcd1d517eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"
Oct 30 13:17:14.855819 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.855701    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3e89b62-a248-46fc-8e83-fc2d4fa66a02-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb\" (UID: \"a3e89b62-a248-46fc-8e83-fc2d4fa66a02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"
Oct 30 13:17:14.856854 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.856709    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3e89b62-a248-46fc-8e83-fc2d4fa66a02-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb\" (UID: \"a3e89b62-a248-46fc-8e83-fc2d4fa66a02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"
Oct 30 13:17:14.881364 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.881059    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgbs5\" (UniqueName: \"kubernetes.io/projected/a3e89b62-a248-46fc-8e83-fc2d4fa66a02-kube-api-access-rgbs5\") pod \"payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb\" (UID: \"a3e89b62-a248-46fc-8e83-fc2d4fa66a02\") " pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"
Oct 30 13:17:14.902703 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.902673    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"
Oct 30 13:17:14.958388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.956758    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a814d069-870d-4ea0-945f-2ddcd1d517eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq\" (UID: \"a814d069-870d-4ea0-945f-2ddcd1d517eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"
Oct 30 13:17:14.958388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.956925    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj48b\" (UniqueName: \"kubernetes.io/projected/a814d069-870d-4ea0-945f-2ddcd1d517eb-kube-api-access-mj48b\") pod \"payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq\" (UID: \"a814d069-870d-4ea0-945f-2ddcd1d517eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"
Oct 30 13:17:14.958388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.958141    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a814d069-870d-4ea0-945f-2ddcd1d517eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq\" (UID: \"a814d069-870d-4ea0-945f-2ddcd1d517eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"
Oct 30 13:17:14.995910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:14.992766    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj48b\" (UniqueName: \"kubernetes.io/projected/a814d069-870d-4ea0-945f-2ddcd1d517eb-kube-api-access-mj48b\") pod \"payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq\" (UID: \"a814d069-870d-4ea0-945f-2ddcd1d517eb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"
Oct 30 13:17:15.103126 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:15.103030    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"
Oct 30 13:17:15.221105 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:15.221071    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"]
Oct 30 13:17:15.242342 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:17:15.224540    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3e89b62_a248_46fc_8e83_fc2d4fa66a02.slice/crio-bb02e8ff5add8c72694287971b7e1e2e6d9764c0f8ff34d8dfa1423a7c992275 WatchSource:0}: Error finding container bb02e8ff5add8c72694287971b7e1e2e6d9764c0f8ff34d8dfa1423a7c992275: Status 404 returned error can't find the container with id bb02e8ff5add8c72694287971b7e1e2e6d9764c0f8ff34d8dfa1423a7c992275
Oct 30 13:17:15.432799 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:15.432697    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"]
Oct 30 13:17:15.438844 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:17:15.438785    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda814d069_870d_4ea0_945f_2ddcd1d517eb.slice/crio-088c7183015c9331b2c88d0a09fcf2a96bee5716e3724ca9ae058b62acba610e WatchSource:0}: Error finding container 088c7183015c9331b2c88d0a09fcf2a96bee5716e3724ca9ae058b62acba610e: Status 404 returned error can't find the container with id 088c7183015c9331b2c88d0a09fcf2a96bee5716e3724ca9ae058b62acba610e
Oct 30 13:17:15.973304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:15.971182    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq" event={"ID":"a814d069-870d-4ea0-945f-2ddcd1d517eb","Type":"ContainerStarted","Data":"6bd17ded77f7b16d969d5d0fd4c01ca24eb89dbaa2400456116506fbefc243df"}
Oct 30 13:17:15.973304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:15.971354    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq" event={"ID":"a814d069-870d-4ea0-945f-2ddcd1d517eb","Type":"ContainerStarted","Data":"088c7183015c9331b2c88d0a09fcf2a96bee5716e3724ca9ae058b62acba610e"}
Oct 30 13:17:15.973304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:15.971392    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"
Oct 30 13:17:15.993297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:15.992959    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb" event={"ID":"a3e89b62-a248-46fc-8e83-fc2d4fa66a02","Type":"ContainerStarted","Data":"a40abb11c813733c53c9e99ff4267ed6acb82546b0f8c25b6e9ec6f1d041cce8"}
Oct 30 13:17:15.993297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:15.993028    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb" event={"ID":"a3e89b62-a248-46fc-8e83-fc2d4fa66a02","Type":"ContainerStarted","Data":"bb02e8ff5add8c72694287971b7e1e2e6d9764c0f8ff34d8dfa1423a7c992275"}
Oct 30 13:17:15.993563 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:15.993328    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"
Oct 30 13:17:16.028126 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:16.028063    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq" podStartSLOduration=2.028045716 podStartE2EDuration="2.028045716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:16.024363503 +0000 UTC m=+1877.607793199" watchObservedRunningTime="2026-10-30 13:17:16.028045716 +0000 UTC m=+1877.611475413"
Oct 30 13:17:16.081152 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:16.080933    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb" podStartSLOduration=2.080911866 podStartE2EDuration="2.080911866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:16.08017459 +0000 UTC m=+1877.663604299" watchObservedRunningTime="2026-10-30 13:17:16.080911866 +0000 UTC m=+1877.664341560"
Oct 30 13:17:16.824326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:16.824261    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"
Oct 30 13:17:16.824516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:16.824407    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"
Oct 30 13:17:18.647068 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:17:18.647029    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:17:18.742822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.742791    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"]
Oct 30 13:17:18.747692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.747673    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.774654 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.774631    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2a3e01-openshift-default-dockercfg-6vxcz\""
Oct 30 13:17:18.789207 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.789162    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"]
Oct 30 13:17:18.834997 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.834955    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-podinfo\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.835192 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.835014    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-token\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.835192 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.835177    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2214ddd-9b56-4e64-8346-9f009202f5d4-istiod-ca-cert\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.835320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.835212    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-workload-certs\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.835320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.835238    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-workload-socket\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.835399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.835361    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-credential-socket\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.835451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.835413    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7q42\" (UniqueName: \"kubernetes.io/projected/a2214ddd-9b56-4e64-8346-9f009202f5d4-kube-api-access-h7q42\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.835526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.835506    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-data\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.835632 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.835619    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-envoy\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.937133 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.937045    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2214ddd-9b56-4e64-8346-9f009202f5d4-istiod-ca-cert\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.937133 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.937092    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-workload-certs\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.937410 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.937219    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-workload-socket\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.937410 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.937332    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-credential-socket\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.937410 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.937401    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7q42\" (UniqueName: \"kubernetes.io/projected/a2214ddd-9b56-4e64-8346-9f009202f5d4-kube-api-access-h7q42\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.937583 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.937448    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-data\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.937583 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.937513    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-workload-certs\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.937583 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.937565    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-envoy\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.937687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.937668    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-podinfo\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.937735 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.937719    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-token\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.938034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.937880    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-workload-socket\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.938034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.937927    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-credential-socket\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.938182 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.938096    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-data\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.938334 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.938314    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2214ddd-9b56-4e64-8346-9f009202f5d4-istiod-ca-cert\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.940131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.940106    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-envoy\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.940221 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.940208    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-podinfo\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.965822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.965789    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-token\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:18.965968 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:18.965944    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7q42\" (UniqueName: \"kubernetes.io/projected/a2214ddd-9b56-4e64-8346-9f009202f5d4-kube-api-access-h7q42\") pod \"e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") " pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:19.064462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:19.064409    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:19.401605 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:17:19.401567    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2214ddd_9b56_4e64_8346_9f009202f5d4.slice/crio-30bfda6f1bd86e49d4c76fb8fdb0f675a2a7b0ce3b4b74c57b860465d5ca1d00 WatchSource:0}: Error finding container 30bfda6f1bd86e49d4c76fb8fdb0f675a2a7b0ce3b4b74c57b860465d5ca1d00: Status 404 returned error can't find the container with id 30bfda6f1bd86e49d4c76fb8fdb0f675a2a7b0ce3b4b74c57b860465d5ca1d00
Oct 30 13:17:19.401605 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:19.401571    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"]
Oct 30 13:17:19.404698 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:19.404655    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:17:19.404798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:19.404731    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:17:19.404798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:19.404759    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:17:20.028202 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:20.028166    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw" event={"ID":"a2214ddd-9b56-4e64-8346-9f009202f5d4","Type":"ContainerStarted","Data":"8fc3e5cb0eb298b3e4945f9d134f8e769b5c9e01de5d03bf1f7505515e1ceded"}
Oct 30 13:17:20.028615 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:20.028249    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw" event={"ID":"a2214ddd-9b56-4e64-8346-9f009202f5d4","Type":"ContainerStarted","Data":"30bfda6f1bd86e49d4c76fb8fdb0f675a2a7b0ce3b4b74c57b860465d5ca1d00"}
Oct 30 13:17:20.072297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:20.071541    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:20.075348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:20.074288    5893 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 13:17:20.075348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:20.074343    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw" podUID="a2214ddd-9b56-4e64-8346-9f009202f5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 13:17:20.094836 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:20.094782    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw" podStartSLOduration=2.094766522 podStartE2EDuration="2.094766522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:20.090938009 +0000 UTC m=+1881.674367722" watchObservedRunningTime="2026-10-30 13:17:20.094766522 +0000 UTC m=+1881.678196215"
Oct 30 13:17:21.073448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:21.073405    5893 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 13:17:21.074070 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:21.074037    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw" podUID="a2214ddd-9b56-4e64-8346-9f009202f5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 13:17:22.071299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:22.065681    5893 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 13:17:22.071299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:22.065741    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw" podUID="a2214ddd-9b56-4e64-8346-9f009202f5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 13:17:23.071356 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:23.068677    5893 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 13:17:23.071356 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:23.068740    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw" podUID="a2214ddd-9b56-4e64-8346-9f009202f5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 13:17:24.069297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:24.068697    5893 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 13:17:24.069297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:24.068768    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw" podUID="a2214ddd-9b56-4e64-8346-9f009202f5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 13:17:25.072301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:25.066059    5893 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Oct 30 13:17:25.072301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:25.066127    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw" podUID="a2214ddd-9b56-4e64-8346-9f009202f5d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Oct 30 13:17:26.115968 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:26.115934    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:26.117000 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:26.116969    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:26.117851 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:26.117828    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:17:27.006798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:27.006766    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"
Oct 30 13:17:27.007243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:27.007223    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"
Oct 30 13:17:44.422721 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.422509    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"]
Oct 30 13:17:44.432762 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.432730    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"
Oct 30 13:17:44.444027 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.443996    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"]
Oct 30 13:17:44.457221 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.457187    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bec983-dockercfg-ffhxw\""
Oct 30 13:17:44.457840 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.457816    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bec983\""
Oct 30 13:17:44.501521 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.501492    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"]
Oct 30 13:17:44.506312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.506265    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"
Oct 30 13:17:44.550299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.546311    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr6wh\" (UniqueName: \"kubernetes.io/projected/0db246df-9cb2-4c31-9389-13fb0c8ac395-kube-api-access-nr6wh\") pod \"payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr\" (UID: \"0db246df-9cb2-4c31-9389-13fb0c8ac395\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"
Oct 30 13:17:44.550299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.546659    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db246df-9cb2-4c31-9389-13fb0c8ac395-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr\" (UID: \"0db246df-9cb2-4c31-9389-13fb0c8ac395\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"
Oct 30 13:17:44.679350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.675355    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db246df-9cb2-4c31-9389-13fb0c8ac395-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr\" (UID: \"0db246df-9cb2-4c31-9389-13fb0c8ac395\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"
Oct 30 13:17:44.679597 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.679359    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vwj7\" (UniqueName: \"kubernetes.io/projected/967ac404-7b3d-49bd-9cb5-63e148bb4f9a-kube-api-access-4vwj7\") pod \"payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj\" (UID: \"967ac404-7b3d-49bd-9cb5-63e148bb4f9a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"
Oct 30 13:17:44.679597 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.679518    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr6wh\" (UniqueName: \"kubernetes.io/projected/0db246df-9cb2-4c31-9389-13fb0c8ac395-kube-api-access-nr6wh\") pod \"payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr\" (UID: \"0db246df-9cb2-4c31-9389-13fb0c8ac395\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"
Oct 30 13:17:44.704834 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.683517    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967ac404-7b3d-49bd-9cb5-63e148bb4f9a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj\" (UID: \"967ac404-7b3d-49bd-9cb5-63e148bb4f9a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"
Oct 30 13:17:44.704834 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.698480    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db246df-9cb2-4c31-9389-13fb0c8ac395-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr\" (UID: \"0db246df-9cb2-4c31-9389-13fb0c8ac395\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"
Oct 30 13:17:44.769207 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.769177    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"]
Oct 30 13:17:44.787175 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.787112    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr6wh\" (UniqueName: \"kubernetes.io/projected/0db246df-9cb2-4c31-9389-13fb0c8ac395-kube-api-access-nr6wh\") pod \"payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr\" (UID: \"0db246df-9cb2-4c31-9389-13fb0c8ac395\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"
Oct 30 13:17:44.792066 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.790541    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967ac404-7b3d-49bd-9cb5-63e148bb4f9a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj\" (UID: \"967ac404-7b3d-49bd-9cb5-63e148bb4f9a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"
Oct 30 13:17:44.792066 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.790734    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vwj7\" (UniqueName: \"kubernetes.io/projected/967ac404-7b3d-49bd-9cb5-63e148bb4f9a-kube-api-access-4vwj7\") pod \"payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj\" (UID: \"967ac404-7b3d-49bd-9cb5-63e148bb4f9a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"
Oct 30 13:17:44.795992 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.795898    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967ac404-7b3d-49bd-9cb5-63e148bb4f9a-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj\" (UID: \"967ac404-7b3d-49bd-9cb5-63e148bb4f9a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"
Oct 30 13:17:44.814722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.814693    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vwj7\" (UniqueName: \"kubernetes.io/projected/967ac404-7b3d-49bd-9cb5-63e148bb4f9a-kube-api-access-4vwj7\") pod \"payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj\" (UID: \"967ac404-7b3d-49bd-9cb5-63e148bb4f9a\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"
Oct 30 13:17:44.859400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:44.859368    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"
Oct 30 13:17:45.062331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:45.062256    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"
Oct 30 13:17:45.182962 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:45.180684    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"]
Oct 30 13:17:45.195651 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:17:45.195618    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967ac404_7b3d_49bd_9cb5_63e148bb4f9a.slice/crio-c94eeb1e37b86344996642877d587dd5c0e8dd6ea7169c8678efccd60619ac1c WatchSource:0}: Error finding container c94eeb1e37b86344996642877d587dd5c0e8dd6ea7169c8678efccd60619ac1c: Status 404 returned error can't find the container with id c94eeb1e37b86344996642877d587dd5c0e8dd6ea7169c8678efccd60619ac1c
Oct 30 13:17:45.296460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:45.296428    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj" event={"ID":"967ac404-7b3d-49bd-9cb5-63e148bb4f9a","Type":"ContainerStarted","Data":"c94eeb1e37b86344996642877d587dd5c0e8dd6ea7169c8678efccd60619ac1c"}
Oct 30 13:17:45.381305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:45.381250    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"]
Oct 30 13:17:45.382751 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:17:45.382722    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db246df_9cb2_4c31_9389_13fb0c8ac395.slice/crio-57ace622c0f98727b1f270036c43462b642f90fa9a48cb5220d55393f313bed4 WatchSource:0}: Error finding container 57ace622c0f98727b1f270036c43462b642f90fa9a48cb5220d55393f313bed4: Status 404 returned error can't find the container with id 57ace622c0f98727b1f270036c43462b642f90fa9a48cb5220d55393f313bed4
Oct 30 13:17:46.305341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:46.305301    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj" event={"ID":"967ac404-7b3d-49bd-9cb5-63e148bb4f9a","Type":"ContainerStarted","Data":"6feebe38f4970f02657ce79c2667643f59944006a033407e7e61087a2bbdf596"}
Oct 30 13:17:46.305788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:46.305434    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"
Oct 30 13:17:46.307612 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:46.307580    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr" event={"ID":"0db246df-9cb2-4c31-9389-13fb0c8ac395","Type":"ContainerStarted","Data":"16747b4a97b604f70df517f77ff7bb6f9d51ab478fa89e49d34cd29cf6b4da1a"}
Oct 30 13:17:46.307872 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:46.307675    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr" event={"ID":"0db246df-9cb2-4c31-9389-13fb0c8ac395","Type":"ContainerStarted","Data":"57ace622c0f98727b1f270036c43462b642f90fa9a48cb5220d55393f313bed4"}
Oct 30 13:17:46.307872 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:46.307710    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"
Oct 30 13:17:46.347448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:46.347401    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj" podStartSLOduration=2.347386214 podStartE2EDuration="2.347386214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:46.3465314 +0000 UTC m=+1907.929961093" watchObservedRunningTime="2026-10-30 13:17:46.347386214 +0000 UTC m=+1907.930815910"
Oct 30 13:17:46.387692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:46.387633    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr" podStartSLOduration=2.387619322 podStartE2EDuration="2.387619322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:46.385560377 +0000 UTC m=+1907.968990069" watchObservedRunningTime="2026-10-30 13:17:46.387619322 +0000 UTC m=+1907.971049057"
Oct 30 13:17:48.647039 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:17:48.646994    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:17:57.319658 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:57.319628    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"
Oct 30 13:17:57.320937 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:17:57.320914    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"
Oct 30 13:18:18.681880 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:18:18.681843    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:18:20.789462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:20.789430    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"]
Oct 30 13:18:20.790219 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:20.790182    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng" podUID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerName="istio-proxy" containerID="cri-o://d570f8a6f067838b98627ad1032579dd49244382377eec0b3772ae06e341c54f" gracePeriod=30
Oct 30 13:18:26.121248 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.120936    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:18:26.254850 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.254770    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-credential-socket\") pod \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") "
Oct 30 13:18:26.255050 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.254895    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istiod-ca-cert\") pod \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") "
Oct 30 13:18:26.255050 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.254939    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-envoy\") pod \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") "
Oct 30 13:18:26.255050 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.255017    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-data\") pod \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") "
Oct 30 13:18:26.255246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.255109    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-credential-socket" pod "d716f5ae-5dba-41f1-aeef-1d4fdeadba33" (UID: "d716f5ae-5dba-41f1-aeef-1d4fdeadba33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:26.255246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.255132    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-kube-api-access-pb69n\" (UniqueName: \"kubernetes.io/projected/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-kube-api-access-pb69n\") pod \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") "
Oct 30 13:18:26.255246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.255175    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-workload-certs\") pod \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") "
Oct 30 13:18:26.255439 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.255244    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-token\" (UniqueName: \"kubernetes.io/projected/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-token\") pod \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") "
Oct 30 13:18:26.255439 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.255298    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-podinfo\") pod \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") "
Oct 30 13:18:26.255439 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.255359    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istiod-ca-cert" pod "d716f5ae-5dba-41f1-aeef-1d4fdeadba33" (UID: "d716f5ae-5dba-41f1-aeef-1d4fdeadba33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:26.255439 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.255398    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-workload-socket\") pod \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\" (UID: \"d716f5ae-5dba-41f1-aeef-1d4fdeadba33\") "
Oct 30 13:18:26.255663 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.255510    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-workload-certs" pod "d716f5ae-5dba-41f1-aeef-1d4fdeadba33" (UID: "d716f5ae-5dba-41f1-aeef-1d4fdeadba33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:26.255720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.255674    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-data" pod "d716f5ae-5dba-41f1-aeef-1d4fdeadba33" (UID: "d716f5ae-5dba-41f1-aeef-1d4fdeadba33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:26.255899 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.255877    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-workload-socket" pod "d716f5ae-5dba-41f1-aeef-1d4fdeadba33" (UID: "d716f5ae-5dba-41f1-aeef-1d4fdeadba33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:26.256250 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.256112    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:18:26.256418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.256265    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:18:26.256418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.256302    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:18:26.256418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.256316    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:18:26.257655 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.257627    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-envoy" pod "d716f5ae-5dba-41f1-aeef-1d4fdeadba33" (UID: "d716f5ae-5dba-41f1-aeef-1d4fdeadba33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:18:26.257813 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.257795    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-token" pod "d716f5ae-5dba-41f1-aeef-1d4fdeadba33" (UID: "d716f5ae-5dba-41f1-aeef-1d4fdeadba33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:26.258217 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.258199    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-kube-api-access-pb69n" pod "d716f5ae-5dba-41f1-aeef-1d4fdeadba33" (UID: "d716f5ae-5dba-41f1-aeef-1d4fdeadba33"). InnerVolumeSpecName "kube-api-access-pb69n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:26.258814 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.258791    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-podinfo" pod "d716f5ae-5dba-41f1-aeef-1d4fdeadba33" (UID: "d716f5ae-5dba-41f1-aeef-1d4fdeadba33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:18:26.357811 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.357736    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb69n\" (UniqueName: \"kubernetes.io/projected/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-kube-api-access-pb69n\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:18:26.357811 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.357764    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:18:26.357811 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.357774    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:18:26.357811 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.357782    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:18:26.357811 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.357791    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d716f5ae-5dba-41f1-aeef-1d4fdeadba33-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:18:26.693902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.693872    5893 generic.go:356] "Generic (PLEG): container finished" podID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" containerID="c36824bd757c6bdd5c0783ba26bc38c0f734b8a1017fac5f9053699202edd58c" exitCode=0
Oct 30 13:18:26.694115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.693947    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb" event={"ID":"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611","Type":"ContainerDied","Data":"c36824bd757c6bdd5c0783ba26bc38c0f734b8a1017fac5f9053699202edd58c"}
Oct 30 13:18:26.694485 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.694469    5893 scope.go:122] "RemoveContainer" containerID="c36824bd757c6bdd5c0783ba26bc38c0f734b8a1017fac5f9053699202edd58c"
Oct 30 13:18:26.696018 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.695993    5893 generic.go:356] "Generic (PLEG): container finished" podID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerID="d570f8a6f067838b98627ad1032579dd49244382377eec0b3772ae06e341c54f" exitCode=0
Oct 30 13:18:26.696132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.696054    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"
Oct 30 13:18:26.696132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.696073    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng" event={"ID":"d716f5ae-5dba-41f1-aeef-1d4fdeadba33","Type":"ContainerDied","Data":"d570f8a6f067838b98627ad1032579dd49244382377eec0b3772ae06e341c54f"}
Oct 30 13:18:26.696132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.696123    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng" event={"ID":"d716f5ae-5dba-41f1-aeef-1d4fdeadba33","Type":"ContainerDied","Data":"b24d5ad44f41475ff3a63937703f9b9a9ef715b71191ee3c7871b5d530f0d50c"}
Oct 30 13:18:26.696236 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.696153    5893 scope.go:122] "RemoveContainer" containerID="d570f8a6f067838b98627ad1032579dd49244382377eec0b3772ae06e341c54f"
Oct 30 13:18:26.696587 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.696561    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:26.696749 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.696637    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:26.696749 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.696679    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:26.712622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.712598    5893 scope.go:122] "RemoveContainer" containerID="207ecee35167d3224270e421a47d925b1f95488a9ecd3ebc0cc9e037e32f2b68"
Oct 30 13:18:26.725169 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.725149    5893 scope.go:122] "RemoveContainer" containerID="d570f8a6f067838b98627ad1032579dd49244382377eec0b3772ae06e341c54f"
Oct 30 13:18:26.725501 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:18:26.725477    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d570f8a6f067838b98627ad1032579dd49244382377eec0b3772ae06e341c54f\": container with ID starting with d570f8a6f067838b98627ad1032579dd49244382377eec0b3772ae06e341c54f not found: ID does not exist" containerID="d570f8a6f067838b98627ad1032579dd49244382377eec0b3772ae06e341c54f"
Oct 30 13:18:26.725585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.725516    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d570f8a6f067838b98627ad1032579dd49244382377eec0b3772ae06e341c54f"} err="failed to get container status \"d570f8a6f067838b98627ad1032579dd49244382377eec0b3772ae06e341c54f\": rpc error: code = NotFound desc = could not find container \"d570f8a6f067838b98627ad1032579dd49244382377eec0b3772ae06e341c54f\": container with ID starting with d570f8a6f067838b98627ad1032579dd49244382377eec0b3772ae06e341c54f not found: ID does not exist"
Oct 30 13:18:26.725585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.725539    5893 scope.go:122] "RemoveContainer" containerID="207ecee35167d3224270e421a47d925b1f95488a9ecd3ebc0cc9e037e32f2b68"
Oct 30 13:18:26.725916 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:18:26.725893    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"207ecee35167d3224270e421a47d925b1f95488a9ecd3ebc0cc9e037e32f2b68\": container with ID starting with 207ecee35167d3224270e421a47d925b1f95488a9ecd3ebc0cc9e037e32f2b68 not found: ID does not exist" containerID="207ecee35167d3224270e421a47d925b1f95488a9ecd3ebc0cc9e037e32f2b68"
Oct 30 13:18:26.725983 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.725926    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"207ecee35167d3224270e421a47d925b1f95488a9ecd3ebc0cc9e037e32f2b68"} err="failed to get container status \"207ecee35167d3224270e421a47d925b1f95488a9ecd3ebc0cc9e037e32f2b68\": rpc error: code = NotFound desc = could not find container \"207ecee35167d3224270e421a47d925b1f95488a9ecd3ebc0cc9e037e32f2b68\": container with ID starting with 207ecee35167d3224270e421a47d925b1f95488a9ecd3ebc0cc9e037e32f2b68 not found: ID does not exist"
Oct 30 13:18:26.792142 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.792097    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"]
Oct 30 13:18:26.800319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:26.800255    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-14e8a4-openshift-default-5d6dcf965-z6qng"]
Oct 30 13:18:27.707214 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:27.707173    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb" event={"ID":"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611","Type":"ContainerStarted","Data":"87036c112675f046b03277fa563ee85cc72108f3014eec0f2b5bfe4d6ff61995"}
Oct 30 13:18:28.113386 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.113353    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"]
Oct 30 13:18:28.114202 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.114186    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerName="istio-proxy"
Oct 30 13:18:28.114260 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.114205    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerName="istio-proxy"
Oct 30 13:18:28.114387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.114375    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerName="istio-proxy"
Oct 30 13:18:28.114427 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.114393    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerName="istio-proxy"
Oct 30 13:18:28.117035 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.117021    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"
Oct 30 13:18:28.155310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.154688    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-60a9a6\""
Oct 30 13:18:28.155310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.154813    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-60a9a6-dockercfg-92x2z\""
Oct 30 13:18:28.199300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.169108    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"]
Oct 30 13:18:28.199300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.193672    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f991025c-c7ba-4b0d-8676-9322d7ba6e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw\" (UID: \"f991025c-c7ba-4b0d-8676-9322d7ba6e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"
Oct 30 13:18:28.199300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.193762    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvnsn\" (UniqueName: \"kubernetes.io/projected/f991025c-c7ba-4b0d-8676-9322d7ba6e7f-kube-api-access-jvnsn\") pod \"payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw\" (UID: \"f991025c-c7ba-4b0d-8676-9322d7ba6e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"
Oct 30 13:18:28.232300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.218648    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"]
Oct 30 13:18:28.232300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.220033    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerName="istio-proxy"
Oct 30 13:18:28.232300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.220059    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" containerName="istio-proxy"
Oct 30 13:18:28.313295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.312555    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"
Oct 30 13:18:28.373353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.372808    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:18:28.403299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.373607    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f991025c-c7ba-4b0d-8676-9322d7ba6e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw\" (UID: \"f991025c-c7ba-4b0d-8676-9322d7ba6e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"
Oct 30 13:18:28.403299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.373718    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvnsn\" (UniqueName: \"kubernetes.io/projected/f991025c-c7ba-4b0d-8676-9322d7ba6e7f-kube-api-access-jvnsn\") pod \"payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw\" (UID: \"f991025c-c7ba-4b0d-8676-9322d7ba6e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"
Oct 30 13:18:28.403299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.374045    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f991025c-c7ba-4b0d-8676-9322d7ba6e7f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw\" (UID: \"f991025c-c7ba-4b0d-8676-9322d7ba6e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"
Oct 30 13:18:28.403299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.385559    5893 patch_prober.go:29] interesting pod/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:18:28.403299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.385632    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb" podUID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 13:18:28.429330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.413478    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"]
Oct 30 13:18:28.429330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.427354    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvnsn\" (UniqueName: \"kubernetes.io/projected/f991025c-c7ba-4b0d-8676-9322d7ba6e7f-kube-api-access-jvnsn\") pod \"payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw\" (UID: \"f991025c-c7ba-4b0d-8676-9322d7ba6e7f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"
Oct 30 13:18:28.431907 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.431655    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"
Oct 30 13:18:28.483753 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.483701    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450ff90c-49e7-45c0-8e3a-6b684aedbd81-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql\" (UID: \"450ff90c-49e7-45c0-8e3a-6b684aedbd81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"
Oct 30 13:18:28.483929 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.483832    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pxb8\" (UniqueName: \"kubernetes.io/projected/450ff90c-49e7-45c0-8e3a-6b684aedbd81-kube-api-access-8pxb8\") pod \"payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql\" (UID: \"450ff90c-49e7-45c0-8e3a-6b684aedbd81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"
Oct 30 13:18:28.537780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.537747    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d716f5ae-5dba-41f1-aeef-1d4fdeadba33" path="/var/lib/kubelet/pods/d716f5ae-5dba-41f1-aeef-1d4fdeadba33/volumes"
Oct 30 13:18:28.585105 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.585065    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pxb8\" (UniqueName: \"kubernetes.io/projected/450ff90c-49e7-45c0-8e3a-6b684aedbd81-kube-api-access-8pxb8\") pod \"payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql\" (UID: \"450ff90c-49e7-45c0-8e3a-6b684aedbd81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"
Oct 30 13:18:28.585753 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.585732    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450ff90c-49e7-45c0-8e3a-6b684aedbd81-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql\" (UID: \"450ff90c-49e7-45c0-8e3a-6b684aedbd81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"
Oct 30 13:18:28.587095 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.586638    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450ff90c-49e7-45c0-8e3a-6b684aedbd81-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql\" (UID: \"450ff90c-49e7-45c0-8e3a-6b684aedbd81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"
Oct 30 13:18:28.613431 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.613405    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pxb8\" (UniqueName: \"kubernetes.io/projected/450ff90c-49e7-45c0-8e3a-6b684aedbd81-kube-api-access-8pxb8\") pod \"payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql\" (UID: \"450ff90c-49e7-45c0-8e3a-6b684aedbd81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"
Oct 30 13:18:28.695839 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.695763    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"
Oct 30 13:18:28.829260 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:28.829205    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"]
Oct 30 13:18:28.832750 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:18:28.832716    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf991025c_c7ba_4b0d_8676_9322d7ba6e7f.slice/crio-56c5c5cb32260a574dec37bdd11a790e10d09d4bc8516b2c32c035274a0747a3 WatchSource:0}: Error finding container 56c5c5cb32260a574dec37bdd11a790e10d09d4bc8516b2c32c035274a0747a3: Status 404 returned error can't find the container with id 56c5c5cb32260a574dec37bdd11a790e10d09d4bc8516b2c32c035274a0747a3
Oct 30 13:18:29.077396 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:18:29.076085    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450ff90c_49e7_45c0_8e3a_6b684aedbd81.slice/crio-b2db48117453ed9b9864d6b8c84e9f667dc5426f335991fc9afb22f57bc5d37b WatchSource:0}: Error finding container b2db48117453ed9b9864d6b8c84e9f667dc5426f335991fc9afb22f57bc5d37b: Status 404 returned error can't find the container with id b2db48117453ed9b9864d6b8c84e9f667dc5426f335991fc9afb22f57bc5d37b
Oct 30 13:18:29.078546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:29.078521    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"]
Oct 30 13:18:29.366359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:29.357922    5893 patch_prober.go:29] interesting pod/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:18:29.366359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:29.358436    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb" podUID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 13:18:29.780977 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:29.780932    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql" event={"ID":"450ff90c-49e7-45c0-8e3a-6b684aedbd81","Type":"ContainerStarted","Data":"078624b008fb14aa405fdd3d598eef7232cb46a2df734a3363e57d6bdd4c134c"}
Oct 30 13:18:29.781313 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:29.781254    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql" event={"ID":"450ff90c-49e7-45c0-8e3a-6b684aedbd81","Type":"ContainerStarted","Data":"b2db48117453ed9b9864d6b8c84e9f667dc5426f335991fc9afb22f57bc5d37b"}
Oct 30 13:18:29.781502 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:29.781491    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"
Oct 30 13:18:29.789267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:29.789238    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw" event={"ID":"f991025c-c7ba-4b0d-8676-9322d7ba6e7f","Type":"ContainerStarted","Data":"f3e5568e05dd8af7a349d26a7cf00e532587096ba30cb38d1c06696a8ebf4851"}
Oct 30 13:18:29.789557 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:29.789538    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw" event={"ID":"f991025c-c7ba-4b0d-8676-9322d7ba6e7f","Type":"ContainerStarted","Data":"56c5c5cb32260a574dec37bdd11a790e10d09d4bc8516b2c32c035274a0747a3"}
Oct 30 13:18:29.789691 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:29.789678    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"
Oct 30 13:18:29.823461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:29.823414    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql" podStartSLOduration=1.823400702 podStartE2EDuration="1.823400702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:29.821676974 +0000 UTC m=+1951.405106667" watchObservedRunningTime="2026-10-30 13:18:29.823400702 +0000 UTC m=+1951.406830395"
Oct 30 13:18:29.870293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:29.869899    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw" podStartSLOduration=1.869879587 podStartE2EDuration="1.869879587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:29.869686964 +0000 UTC m=+1951.453116658" watchObservedRunningTime="2026-10-30 13:18:29.869879587 +0000 UTC m=+1951.453309282"
Oct 30 13:18:30.361299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:30.357678    5893 patch_prober.go:29] interesting pod/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:18:30.361299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:30.357741    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb" podUID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 13:18:30.766299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:30.757120    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:18:31.358703 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:31.358668    5893 patch_prober.go:29] interesting pod/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:18:31.359342 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:31.359311    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb" podUID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 13:18:31.515722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:31.515690    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"]
Oct 30 13:18:31.515929 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:31.515909    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79" podUID="e16875e0-567d-4a21-916e-9d1338d15e14" containerName="maas-api" containerID="cri-o://3a6e6263ed62b01e3ddbe1d171d9a1cc9b270b52fb0530d29d368f1f1b27ce20" gracePeriod=30
Oct 30 13:18:31.851310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:31.848507    5893 generic.go:356] "Generic (PLEG): container finished" podID="e16875e0-567d-4a21-916e-9d1338d15e14" containerID="3a6e6263ed62b01e3ddbe1d171d9a1cc9b270b52fb0530d29d368f1f1b27ce20" exitCode=0
Oct 30 13:18:31.851310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:31.849567    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79" event={"ID":"e16875e0-567d-4a21-916e-9d1338d15e14","Type":"ContainerDied","Data":"3a6e6263ed62b01e3ddbe1d171d9a1cc9b270b52fb0530d29d368f1f1b27ce20"}
Oct 30 13:18:32.228625 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.227576    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:18:32.313570 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.313478    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e16875e0-567d-4a21-916e-9d1338d15e14-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16875e0-567d-4a21-916e-9d1338d15e14-metrics-certs\") pod \"e16875e0-567d-4a21-916e-9d1338d15e14\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") "
Oct 30 13:18:32.314005 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.313983    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e16875e0-567d-4a21-916e-9d1338d15e14-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16875e0-567d-4a21-916e-9d1338d15e14-openshift-service-ca\") pod \"e16875e0-567d-4a21-916e-9d1338d15e14\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") "
Oct 30 13:18:32.314402 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.314375    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e16875e0-567d-4a21-916e-9d1338d15e14-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16875e0-567d-4a21-916e-9d1338d15e14-maas-api-tls\") pod \"e16875e0-567d-4a21-916e-9d1338d15e14\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") "
Oct 30 13:18:32.314629 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.314610    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e16875e0-567d-4a21-916e-9d1338d15e14-kube-api-access-z7gzj\" (UniqueName: \"kubernetes.io/projected/e16875e0-567d-4a21-916e-9d1338d15e14-kube-api-access-z7gzj\") pod \"e16875e0-567d-4a21-916e-9d1338d15e14\" (UID: \"e16875e0-567d-4a21-916e-9d1338d15e14\") "
Oct 30 13:18:32.315100 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.315073    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e16875e0-567d-4a21-916e-9d1338d15e14-openshift-service-ca" pod "e16875e0-567d-4a21-916e-9d1338d15e14" (UID: "e16875e0-567d-4a21-916e-9d1338d15e14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:32.315743 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.315721    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e16875e0-567d-4a21-916e-9d1338d15e14-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:18:32.316872 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.316849    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16875e0-567d-4a21-916e-9d1338d15e14-maas-api-tls" pod "e16875e0-567d-4a21-916e-9d1338d15e14" (UID: "e16875e0-567d-4a21-916e-9d1338d15e14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:32.317398 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.317378    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e16875e0-567d-4a21-916e-9d1338d15e14-kube-api-access-z7gzj" pod "e16875e0-567d-4a21-916e-9d1338d15e14" (UID: "e16875e0-567d-4a21-916e-9d1338d15e14"). InnerVolumeSpecName "kube-api-access-z7gzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:32.317546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.317398    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e16875e0-567d-4a21-916e-9d1338d15e14-metrics-certs" pod "e16875e0-567d-4a21-916e-9d1338d15e14" (UID: "e16875e0-567d-4a21-916e-9d1338d15e14"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:32.379782 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.379749    5893 patch_prober.go:29] interesting pod/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Oct 30 13:18:32.380387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.380357    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb" podUID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Oct 30 13:18:32.386346 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.386206    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"]
Oct 30 13:18:32.387517 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.387492    5893 cpu_manager.go:414] "containerMap: removing container" podUID="e16875e0-567d-4a21-916e-9d1338d15e14" containerName="maas-api"
Oct 30 13:18:32.387517 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.387518    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e16875e0-567d-4a21-916e-9d1338d15e14" containerName="maas-api"
Oct 30 13:18:32.387824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.387798    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="e16875e0-567d-4a21-916e-9d1338d15e14" containerName="maas-api"
Oct 30 13:18:32.391047 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.391030    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:32.402506 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.401696    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"]
Oct 30 13:18:32.411546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.409652    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-bec983-serving-cert\""
Oct 30 13:18:32.424573 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.419873    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e16875e0-567d-4a21-916e-9d1338d15e14-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:18:32.424573 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.419902    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e16875e0-567d-4a21-916e-9d1338d15e14-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:18:32.424573 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.419917    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7gzj\" (UniqueName: \"kubernetes.io/projected/e16875e0-567d-4a21-916e-9d1338d15e14-kube-api-access-z7gzj\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:18:32.534986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.521101    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-maas-api-tls\") pod \"maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:32.534986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.521170    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-metrics-certs\") pod \"maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:32.534986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.522252    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbnkg\" (UniqueName: \"kubernetes.io/projected/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-kube-api-access-pbnkg\") pod \"maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:32.534986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.522425    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:32.624438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.624400    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbnkg\" (UniqueName: \"kubernetes.io/projected/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-kube-api-access-pbnkg\") pod \"maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:32.624647 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.624532    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:32.625129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.625101    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-openshift-service-ca\") pod \"maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:32.625379 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.625346    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-maas-api-tls\") pod \"maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:32.625480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.625432    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-metrics-certs\") pod \"maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:32.628014 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.627988    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-metrics-certs\") pod \"maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:32.628102 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.628044    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-maas-api-tls\") pod \"maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:32.682298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.682064    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbnkg\" (UniqueName: \"kubernetes.io/projected/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-kube-api-access-pbnkg\") pod \"maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:32.711832 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.711652    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:32.881348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.879327    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79" event={"ID":"e16875e0-567d-4a21-916e-9d1338d15e14","Type":"ContainerDied","Data":"4b5165b2226120fa722d0fa0bb75c481524dc413a7de081449fb7265135ce3e9"}
Oct 30 13:18:32.881348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.879402    5893 scope.go:122] "RemoveContainer" containerID="3a6e6263ed62b01e3ddbe1d171d9a1cc9b270b52fb0530d29d368f1f1b27ce20"
Oct 30 13:18:32.881348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.879559    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"
Oct 30 13:18:32.935654 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.935620    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"]
Oct 30 13:18:32.948588 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:32.948555    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-11cfa6-58bd89db75-lfv79"]
Oct 30 13:18:33.086593 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:33.086567    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"]
Oct 30 13:18:33.089491 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:18:33.089455    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40116e5b_e2b3_45b8_93e9_a0faa7fbe950.slice/crio-c5ddeae0b68c6498feb24b5afe94d200aeece109b15dc39bdd85f72e383931fb WatchSource:0}: Error finding container c5ddeae0b68c6498feb24b5afe94d200aeece109b15dc39bdd85f72e383931fb: Status 404 returned error can't find the container with id c5ddeae0b68c6498feb24b5afe94d200aeece109b15dc39bdd85f72e383931fb
Oct 30 13:18:33.091672 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:33.091651    5893 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:18:33.362334 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:33.362228    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:18:33.363571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:33.363549    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:18:33.893164 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:33.893082    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc" event={"ID":"40116e5b-e2b3-45b8-93e9-a0faa7fbe950","Type":"ContainerStarted","Data":"4a647f6fbffa7fd77823e0e7b4068ae4d6fb058dfc15808c0b327bfca0ae5431"}
Oct 30 13:18:33.893164 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:33.893142    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc" event={"ID":"40116e5b-e2b3-45b8-93e9-a0faa7fbe950","Type":"ContainerStarted","Data":"c5ddeae0b68c6498feb24b5afe94d200aeece109b15dc39bdd85f72e383931fb"}
Oct 30 13:18:33.893647 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:33.893176    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:33.936808 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:33.936746    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc" podStartSLOduration=1.507707163 podStartE2EDuration="1.936727686s" totalImagesPullingTime="429.020523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:33.934558735 +0000 UTC m=+1955.517988465" watchObservedRunningTime="2026-10-30 13:18:33.936727686 +0000 UTC m=+1955.520157381"
Oct 30 13:18:34.533963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:34.533921    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e16875e0-567d-4a21-916e-9d1338d15e14" path="/var/lib/kubelet/pods/e16875e0-567d-4a21-916e-9d1338d15e14/volumes"
Oct 30 13:18:39.753788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.753754    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"]
Oct 30 13:18:39.759325 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.759260    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.794327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.786775    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6f9071-openshift-default-dockercfg-vvqfz\""
Oct 30 13:18:39.794327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.794005    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"]
Oct 30 13:18:39.836432 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.836391    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-data\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.836432 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.836432    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istiod-ca-cert\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.836681 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.836449    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-podinfo\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.836681 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.836633    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-credential-socket\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.836681 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.836670    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-workload-socket\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.836853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.836688    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-workload-certs\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.836853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.836743    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-envoy\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.836957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.836877    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x6nw\" (UniqueName: \"kubernetes.io/projected/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-kube-api-access-4x6nw\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.837037 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.837023    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-token\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.909233 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.909204    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:18:39.938759 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.938726    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x6nw\" (UniqueName: \"kubernetes.io/projected/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-kube-api-access-4x6nw\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.938949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.938873    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-token\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.939052 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.939034    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-data\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.939146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.939068    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istiod-ca-cert\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.939146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.939097    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-podinfo\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.939286 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.939189    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-credential-socket\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.939286 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.939235    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-workload-socket\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.939286 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.939260    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-workload-certs\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.939470 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.939309    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-envoy\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.939798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.939764    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-credential-socket\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.939952 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.939867    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istiod-ca-cert\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.939952 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.939934    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-workload-socket\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.940216 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.940195    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-workload-certs\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.940355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.940329    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-data\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.941763 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.941741    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-envoy\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.942194 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.942173    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-podinfo\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.964894 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.964862    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x6nw\" (UniqueName: \"kubernetes.io/projected/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-kube-api-access-4x6nw\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:39.974518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:39.974450    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-token\") pod \"e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") " pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:40.075215 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:40.075128    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:40.437533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:40.436341    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"]
Oct 30 13:18:40.439583 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:18:40.439556    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079c739c_702b_4eeb_9e5e_b30ad98b8fd2.slice/crio-89b4e30a46d015190c7049bda029aa6b66db2e756f362a054cdc0575630b4c49 WatchSource:0}: Error finding container 89b4e30a46d015190c7049bda029aa6b66db2e756f362a054cdc0575630b4c49: Status 404 returned error can't find the container with id 89b4e30a46d015190c7049bda029aa6b66db2e756f362a054cdc0575630b4c49
Oct 30 13:18:40.446653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:40.446614    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:40.446759 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:40.446712    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:40.446797 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:40.446753    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:40.802474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:40.802445    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"
Oct 30 13:18:40.984989 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:40.969476    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr" event={"ID":"079c739c-702b-4eeb-9e5e-b30ad98b8fd2","Type":"ContainerStarted","Data":"117b0791438fc3c50cac6b93a3a18acbc2475f1d1cad45d984e16e7005eac806"}
Oct 30 13:18:40.984989 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:40.969543    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr" event={"ID":"079c739c-702b-4eeb-9e5e-b30ad98b8fd2","Type":"ContainerStarted","Data":"89b4e30a46d015190c7049bda029aa6b66db2e756f362a054cdc0575630b4c49"}
Oct 30 13:18:41.032980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.032904    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr" podStartSLOduration=2.032882286 podStartE2EDuration="2.032882286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:41.031550649 +0000 UTC m=+1962.614980369" watchObservedRunningTime="2026-10-30 13:18:41.032882286 +0000 UTC m=+1962.616311981"
Oct 30 13:18:41.076110 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.076022    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:41.078027 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.078000    5893 patch_prober.go:29] interesting pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:18:41.078180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.078051    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr" podUID="079c739c-702b-4eeb-9e5e-b30ad98b8fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 13:18:41.801340 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.801212    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"]
Oct 30 13:18:41.815293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.812743    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:41.829556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.829519    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"]
Oct 30 13:18:41.835180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.833407    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2b8461-openshift-default-dockercfg-rpkjj\""
Oct 30 13:18:41.889456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.889425    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"
Oct 30 13:18:41.916534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.914830    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26375037-b209-4e8f-8dec-821f2a597ffb-istio-token\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:41.916534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.914895    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-workload-socket\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:41.916534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.914933    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26375037-b209-4e8f-8dec-821f2a597ffb-istio-podinfo\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:41.916534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.914989    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-workload-certs\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:41.916534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.915045    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-istio-envoy\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:41.916534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.915091    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26375037-b209-4e8f-8dec-821f2a597ffb-istiod-ca-cert\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:41.916534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.915217    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-istio-data\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:41.916534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.915403    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-credential-socket\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:41.916534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:41.915465    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhk5s\" (UniqueName: \"kubernetes.io/projected/26375037-b209-4e8f-8dec-821f2a597ffb-kube-api-access-vhk5s\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.026536 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.026495    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-credential-socket\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.026808 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.026783    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhk5s\" (UniqueName: \"kubernetes.io/projected/26375037-b209-4e8f-8dec-821f2a597ffb-kube-api-access-vhk5s\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.026922 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.026905    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26375037-b209-4e8f-8dec-821f2a597ffb-istio-token\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.026986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.026950    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-workload-socket\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.027041 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.026986    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26375037-b209-4e8f-8dec-821f2a597ffb-istio-podinfo\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.027098 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.027048    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-workload-certs\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.027150 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.027110    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-istio-envoy\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.027206 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.027161    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26375037-b209-4e8f-8dec-821f2a597ffb-istiod-ca-cert\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.027341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.027324    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-istio-data\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.027979 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.027953    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-credential-socket\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.029988 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.028979    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-workload-socket\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.033067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.032095    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-workload-certs\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.033067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.032405    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-istio-data\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.033067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.032781    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26375037-b209-4e8f-8dec-821f2a597ffb-istiod-ca-cert\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.033067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.032881    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26375037-b209-4e8f-8dec-821f2a597ffb-istio-podinfo\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.034060 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.034012    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-istio-envoy\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.078374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.075254    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26375037-b209-4e8f-8dec-821f2a597ffb-istio-token\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.078374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.077187    5893 patch_prober.go:29] interesting pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:18:42.078374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.077260    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr" podUID="079c739c-702b-4eeb-9e5e-b30ad98b8fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 13:18:42.081902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.081876    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhk5s\" (UniqueName: \"kubernetes.io/projected/26375037-b209-4e8f-8dec-821f2a597ffb-kube-api-access-vhk5s\") pod \"e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") " pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.191627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.191590    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:42.591380 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.591342    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"]
Oct 30 13:18:42.707756 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.706936    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:42.707756 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.707036    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:42.707756 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:42.707077    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:43.064091 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:43.063992    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" event={"ID":"26375037-b209-4e8f-8dec-821f2a597ffb","Type":"ContainerStarted","Data":"3097b6426a10e870bbd3eaf255bec1bda747cfbff9f37057ce164ab8cdac4e6f"}
Oct 30 13:18:43.064091 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:43.064050    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" event={"ID":"26375037-b209-4e8f-8dec-821f2a597ffb","Type":"ContainerStarted","Data":"aa21329cce5aa388637740982b66aafb4b30709843f08c020d3b98ff0367da28"}
Oct 30 13:18:43.091225 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:43.090263    5893 patch_prober.go:29] interesting pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:18:43.091506 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:43.091483    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr" podUID="079c739c-702b-4eeb-9e5e-b30ad98b8fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 13:18:43.115514 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:43.115424    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" podStartSLOduration=2.115408857 podStartE2EDuration="2.115408857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:43.111584711 +0000 UTC m=+1964.695014426" watchObservedRunningTime="2026-10-30 13:18:43.115408857 +0000 UTC m=+1964.698838549"
Oct 30 13:18:43.193682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:43.192705    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:43.194384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:43.194351    5893 patch_prober.go:29] interesting pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:18:43.194502 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:43.194403    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 13:18:44.077729 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:44.077687    5893 patch_prober.go:29] interesting pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:18:44.078181 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:44.077748    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr" podUID="079c739c-702b-4eeb-9e5e-b30ad98b8fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 13:18:44.198730 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:44.198703    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:45.081630 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:45.078072    5893 patch_prober.go:29] interesting pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:18:45.081630 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:45.078146    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr" podUID="079c739c-702b-4eeb-9e5e-b30ad98b8fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 13:18:45.099192 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:45.099161    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:45.100236 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:45.100213    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:18:46.075815 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:46.075766    5893 patch_prober.go:29] interesting pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:18:46.076005 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:46.075848    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr" podUID="079c739c-702b-4eeb-9e5e-b30ad98b8fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 13:18:47.076314 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:47.076245    5893 patch_prober.go:29] interesting pod/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body=
Oct 30 13:18:47.076801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:47.076352    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr" podUID="079c739c-702b-4eeb-9e5e-b30ad98b8fd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused"
Oct 30 13:18:48.092419 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:48.092383    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:48.094369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:48.094341    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:48.095221 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:48.095204    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:18:48.971301 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:18:48.964468    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:18:56.892471 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:56.892431    5893 patch_prober.go:29] interesting pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:18:56.892938 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:56.892518    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:58.248429 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:58.248397    5893 generic.go:356] "Generic (PLEG): container finished" podID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerID="18234e525c3631c355702e50f4a6dc0e387feca51f1d515e4c4bba2bcdebc11f" exitCode=0
Oct 30 13:18:58.248923 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:58.248444    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" event={"ID":"68b6b218-4523-481c-8f22-b05a6e80af0a","Type":"ContainerDied","Data":"18234e525c3631c355702e50f4a6dc0e387feca51f1d515e4c4bba2bcdebc11f"}
Oct 30 13:18:58.248923 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:58.248885    5893 scope.go:122] "RemoveContainer" containerID="18234e525c3631c355702e50f4a6dc0e387feca51f1d515e4c4bba2bcdebc11f"
Oct 30 13:18:58.250965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:58.250936    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:58.251051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:58.251012    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:58.251051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:58.251045    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:18:59.276304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:59.276250    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" event={"ID":"68b6b218-4523-481c-8f22-b05a6e80af0a","Type":"ContainerStarted","Data":"3e0eefdebdfca637da55f08bc21975af817b34a808d928286f5f630d7518fab0"}
Oct 30 13:18:59.489323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:59.487149    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:18:59.489323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:59.487653    5893 patch_prober.go:29] interesting pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 13:18:59.489323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:18:59.487748    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 13:19:00.486954 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:00.486905    5893 patch_prober.go:29] interesting pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 13:19:00.487405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:00.486968    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 13:19:01.485991 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:01.485958    5893 patch_prober.go:29] interesting pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 13:19:01.487063 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:01.487029    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 13:19:02.486695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:02.486644    5893 patch_prober.go:29] interesting pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 13:19:02.486906 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:02.486707    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 13:19:03.486464 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:03.486427    5893 patch_prober.go:29] interesting pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 13:19:03.486919 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:03.486487    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 13:19:04.486637 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:04.486607    5893 patch_prober.go:29] interesting pod/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body=
Oct 30 13:19:04.487041 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:04.486657    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused"
Oct 30 13:19:05.491242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:05.491210    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:19:05.491821 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:05.491794    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:19:05.492567 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:05.492546    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:19:06.363558 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.363525    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"]
Oct 30 13:19:06.368059 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.368038    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"
Oct 30 13:19:06.383787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.383764    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-619b9b\""
Oct 30 13:19:06.383926 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.383850    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-619b9b-dockercfg-fvscr\""
Oct 30 13:19:06.399221 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.399191    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"]
Oct 30 13:19:06.441243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.441208    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"]
Oct 30 13:19:06.446116 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.446089    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"
Oct 30 13:19:06.520296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.473085    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe59aa0-d066-43f1-97de-9fa4775c71bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2\" (UID: \"bfe59aa0-d066-43f1-97de-9fa4775c71bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"
Oct 30 13:19:06.521010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.520978    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbfz\" (UniqueName: \"kubernetes.io/projected/bfe59aa0-d066-43f1-97de-9fa4775c71bf-kube-api-access-6nbfz\") pod \"payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2\" (UID: \"bfe59aa0-d066-43f1-97de-9fa4775c71bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"
Oct 30 13:19:06.688348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.683023    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xq24\" (UniqueName: \"kubernetes.io/projected/41ef6587-02b7-4fa4-9225-d090977e2310-kube-api-access-2xq24\") pod \"payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk\" (UID: \"41ef6587-02b7-4fa4-9225-d090977e2310\") " pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"
Oct 30 13:19:06.688348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.683122    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41ef6587-02b7-4fa4-9225-d090977e2310-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk\" (UID: \"41ef6587-02b7-4fa4-9225-d090977e2310\") " pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"
Oct 30 13:19:06.688348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.683304    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe59aa0-d066-43f1-97de-9fa4775c71bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2\" (UID: \"bfe59aa0-d066-43f1-97de-9fa4775c71bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"
Oct 30 13:19:06.688348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.683385    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbfz\" (UniqueName: \"kubernetes.io/projected/bfe59aa0-d066-43f1-97de-9fa4775c71bf-kube-api-access-6nbfz\") pod \"payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2\" (UID: \"bfe59aa0-d066-43f1-97de-9fa4775c71bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"
Oct 30 13:19:06.688348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.684866    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe59aa0-d066-43f1-97de-9fa4775c71bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2\" (UID: \"bfe59aa0-d066-43f1-97de-9fa4775c71bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"
Oct 30 13:19:06.698308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.697824    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"]
Oct 30 13:19:06.711054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.711022    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbfz\" (UniqueName: \"kubernetes.io/projected/bfe59aa0-d066-43f1-97de-9fa4775c71bf-kube-api-access-6nbfz\") pod \"payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2\" (UID: \"bfe59aa0-d066-43f1-97de-9fa4775c71bf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"
Oct 30 13:19:06.785102 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.785061    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41ef6587-02b7-4fa4-9225-d090977e2310-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk\" (UID: \"41ef6587-02b7-4fa4-9225-d090977e2310\") " pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"
Oct 30 13:19:06.785398 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.785381    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xq24\" (UniqueName: \"kubernetes.io/projected/41ef6587-02b7-4fa4-9225-d090977e2310-kube-api-access-2xq24\") pod \"payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk\" (UID: \"41ef6587-02b7-4fa4-9225-d090977e2310\") " pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"
Oct 30 13:19:06.786021 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.785991    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41ef6587-02b7-4fa4-9225-d090977e2310-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk\" (UID: \"41ef6587-02b7-4fa4-9225-d090977e2310\") " pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"
Oct 30 13:19:06.825713 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.825678    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xq24\" (UniqueName: \"kubernetes.io/projected/41ef6587-02b7-4fa4-9225-d090977e2310-kube-api-access-2xq24\") pod \"payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk\" (UID: \"41ef6587-02b7-4fa4-9225-d090977e2310\") " pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"
Oct 30 13:19:06.879231 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.879199    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"
Oct 30 13:19:06.983683 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:06.983654    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"
Oct 30 13:19:07.204944 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:19:07.204904    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ef6587_02b7_4fa4_9225_d090977e2310.slice/crio-29475e5efa82c287083eb93292b5b62cfbd3264e327ddc197828f37510f2d42d WatchSource:0}: Error finding container 29475e5efa82c287083eb93292b5b62cfbd3264e327ddc197828f37510f2d42d: Status 404 returned error can't find the container with id 29475e5efa82c287083eb93292b5b62cfbd3264e327ddc197828f37510f2d42d
Oct 30 13:19:07.205456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:07.205430    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"]
Oct 30 13:19:07.327612 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:07.327578    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"]
Oct 30 13:19:07.328619 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:19:07.328597    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe59aa0_d066_43f1_97de_9fa4775c71bf.slice/crio-2478f28c7445ef3d07a99c3a7bd2fb07c20c1278bae953ebe732d875638c1ee7 WatchSource:0}: Error finding container 2478f28c7445ef3d07a99c3a7bd2fb07c20c1278bae953ebe732d875638c1ee7: Status 404 returned error can't find the container with id 2478f28c7445ef3d07a99c3a7bd2fb07c20c1278bae953ebe732d875638c1ee7
Oct 30 13:19:07.432185 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:07.432150    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk" event={"ID":"41ef6587-02b7-4fa4-9225-d090977e2310","Type":"ContainerStarted","Data":"519200e0bece3bacdd11a2bc5cd82215171b2d1a93331df2b5913c4b4558f750"}
Oct 30 13:19:07.432370 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:07.432212    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"
Oct 30 13:19:07.432370 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:07.432233    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk" event={"ID":"41ef6587-02b7-4fa4-9225-d090977e2310","Type":"ContainerStarted","Data":"29475e5efa82c287083eb93292b5b62cfbd3264e327ddc197828f37510f2d42d"}
Oct 30 13:19:07.434016 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:07.433991    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2" event={"ID":"bfe59aa0-d066-43f1-97de-9fa4775c71bf","Type":"ContainerStarted","Data":"c18fd38beac63be783eca55f565a520b3f5e78bede233cc24406f2f0c2626f98"}
Oct 30 13:19:07.434143 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:07.434031    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2" event={"ID":"bfe59aa0-d066-43f1-97de-9fa4775c71bf","Type":"ContainerStarted","Data":"2478f28c7445ef3d07a99c3a7bd2fb07c20c1278bae953ebe732d875638c1ee7"}
Oct 30 13:19:07.434143 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:07.434057    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"
Oct 30 13:19:07.488914 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:07.488857    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk" podStartSLOduration=1.488838162 podStartE2EDuration="1.488838162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:07.485847825 +0000 UTC m=+1989.069277547" watchObservedRunningTime="2026-10-30 13:19:07.488838162 +0000 UTC m=+1989.072267860"
Oct 30 13:19:07.543948 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:07.543901    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2" podStartSLOduration=1.543887569 podStartE2EDuration="1.543887569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:07.539986242 +0000 UTC m=+1989.123415950" watchObservedRunningTime="2026-10-30 13:19:07.543887569 +0000 UTC m=+1989.127317262"
Oct 30 13:19:18.445763 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:18.445734    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"
Oct 30 13:19:18.446412 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:18.446059    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"
Oct 30 13:19:18.647956 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:19:18.647917    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:19:48.647924 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:19:48.647882    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:19:59.726782 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:19:59.726749    5893 scope.go:122] "RemoveContainer" containerID="28bb9f875cd3d9dfcca5db15ccecd51caccadc70a15cca12b39b28ab78c4ba82"
Oct 30 13:20:09.048100 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:09.048063    5893 generic.go:356] "Generic (PLEG): container finished" podID="26375037-b209-4e8f-8dec-821f2a597ffb" containerID="3097b6426a10e870bbd3eaf255bec1bda747cfbff9f37057ce164ab8cdac4e6f" exitCode=0
Oct 30 13:20:09.048596 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:09.048128    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" event={"ID":"26375037-b209-4e8f-8dec-821f2a597ffb","Type":"ContainerDied","Data":"3097b6426a10e870bbd3eaf255bec1bda747cfbff9f37057ce164ab8cdac4e6f"}
Oct 30 13:20:09.048685 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:09.048669    5893 scope.go:122] "RemoveContainer" containerID="3097b6426a10e870bbd3eaf255bec1bda747cfbff9f37057ce164ab8cdac4e6f"
Oct 30 13:20:09.051075 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:09.051042    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:20:09.051191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:09.051134    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:20:09.051238 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:09.051203    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:20:09.695244 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:09.695215    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"]
Oct 30 13:20:10.080691 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:10.080661    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" event={"ID":"26375037-b209-4e8f-8dec-821f2a597ffb","Type":"ContainerStarted","Data":"de1b3149dc338493d154c351ab5aa362fdbf4f6be8fa5190202bc7ccf6baab37"}
Oct 30 13:20:10.193364 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:10.193330    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:20:10.194393 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:10.194368    5893 patch_prober.go:29] interesting pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:20:10.194536 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:10.194418    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 13:20:11.193498 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:11.193452    5893 patch_prober.go:29] interesting pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:20:11.193930 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:11.193542    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 13:20:12.192611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:12.192578    5893 patch_prober.go:29] interesting pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:20:12.192835 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:12.192640    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 13:20:13.193400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:13.193360    5893 patch_prober.go:29] interesting pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:20:13.193820 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:13.193426    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 13:20:14.192824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:14.192791    5893 patch_prober.go:29] interesting pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:20:14.193010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:14.192846    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 13:20:14.696214 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:14.696142    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw" podUID="27960fcb-f56f-42e8-8233-430811c533e8" containerName="payload-processing" containerID="cri-o://14e70bc12788b902dd361909bd24ce61462975a653bd2a0078adf971f7c9d0ac" gracePeriod=25
Oct 30 13:20:15.099854 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:15.099810    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:20:15.192866 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:15.192829    5893 patch_prober.go:29] interesting pod/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body=
Oct 30 13:20:15.193059 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:15.192894    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused"
Oct 30 13:20:16.200124 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:16.200086    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:20:16.201417 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:16.201392    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:20:18.647858 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:20:18.647788    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:20:20.377663 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:20.377626    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw" podUID="27960fcb-f56f-42e8-8233-430811c533e8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:20:30.378066 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:30.378029    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw" podUID="27960fcb-f56f-42e8-8233-430811c533e8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:20:32.949557 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:32.949525    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"]
Oct 30 13:20:32.953502 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:32.953484    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"
Oct 30 13:20:32.974653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:32.974624    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"]
Oct 30 13:20:32.981708 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:32.981672    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f3338e\""
Oct 30 13:20:33.059304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:33.058346    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4rtj\" (UniqueName: \"kubernetes.io/projected/c95f65d7-f870-4b93-8899-a82a15419e8e-kube-api-access-g4rtj\") pod \"payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5\" (UID: \"c95f65d7-f870-4b93-8899-a82a15419e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"
Oct 30 13:20:33.059304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:33.058531    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95f65d7-f870-4b93-8899-a82a15419e8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5\" (UID: \"c95f65d7-f870-4b93-8899-a82a15419e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"
Oct 30 13:20:33.060662 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:33.060630    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f3338e-dockercfg-gjrdb\""
Oct 30 13:20:33.204157 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:33.204020    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4rtj\" (UniqueName: \"kubernetes.io/projected/c95f65d7-f870-4b93-8899-a82a15419e8e-kube-api-access-g4rtj\") pod \"payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5\" (UID: \"c95f65d7-f870-4b93-8899-a82a15419e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"
Oct 30 13:20:33.211981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:33.211950    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95f65d7-f870-4b93-8899-a82a15419e8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5\" (UID: \"c95f65d7-f870-4b93-8899-a82a15419e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"
Oct 30 13:20:33.215453 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:33.213618    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95f65d7-f870-4b93-8899-a82a15419e8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5\" (UID: \"c95f65d7-f870-4b93-8899-a82a15419e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"
Oct 30 13:20:33.267067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:33.264082    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4rtj\" (UniqueName: \"kubernetes.io/projected/c95f65d7-f870-4b93-8899-a82a15419e8e-kube-api-access-g4rtj\") pod \"payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5\" (UID: \"c95f65d7-f870-4b93-8899-a82a15419e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"
Oct 30 13:20:33.272297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:33.271889    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"
Oct 30 13:20:33.643632 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:20:33.643596    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95f65d7_f870_4b93_8899_a82a15419e8e.slice/crio-028329e14ccdf8456729f0378fa79f8376f1c1e51ccf5acd54a526051c097ed7 WatchSource:0}: Error finding container 028329e14ccdf8456729f0378fa79f8376f1c1e51ccf5acd54a526051c097ed7: Status 404 returned error can't find the container with id 028329e14ccdf8456729f0378fa79f8376f1c1e51ccf5acd54a526051c097ed7
Oct 30 13:20:33.652956 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:33.652927    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"]
Oct 30 13:20:34.326376 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:34.326335    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5" event={"ID":"c95f65d7-f870-4b93-8899-a82a15419e8e","Type":"ContainerStarted","Data":"2d11427e6baa82eee3c28987ddaefbb4543369a0a50cf2821516e5b6bb6ef9f9"}
Oct 30 13:20:34.326877 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:34.326433    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5" event={"ID":"c95f65d7-f870-4b93-8899-a82a15419e8e","Type":"ContainerStarted","Data":"028329e14ccdf8456729f0378fa79f8376f1c1e51ccf5acd54a526051c097ed7"}
Oct 30 13:20:34.326877 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:34.326573    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"
Oct 30 13:20:34.388992 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:34.388933    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5" podStartSLOduration=2.3889163350000002 podStartE2EDuration="2.388916335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:34.385543197 +0000 UTC m=+2075.968972890" watchObservedRunningTime="2026-10-30 13:20:34.388916335 +0000 UTC m=+2075.972346027"
Oct 30 13:20:35.063651 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.063619    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"
Oct 30 13:20:35.146230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.146198    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27960fcb-f56f-42e8-8233-430811c533e8-kube-api-access-kc9vm\" (UniqueName: \"kubernetes.io/projected/27960fcb-f56f-42e8-8233-430811c533e8-kube-api-access-kc9vm\") pod \"27960fcb-f56f-42e8-8233-430811c533e8\" (UID: \"27960fcb-f56f-42e8-8233-430811c533e8\") "
Oct 30 13:20:35.146965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.146641    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27960fcb-f56f-42e8-8233-430811c533e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27960fcb-f56f-42e8-8233-430811c533e8-plugins-config-volume\") pod \"27960fcb-f56f-42e8-8233-430811c533e8\" (UID: \"27960fcb-f56f-42e8-8233-430811c533e8\") "
Oct 30 13:20:35.147077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.147007    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27960fcb-f56f-42e8-8233-430811c533e8-plugins-config-volume" pod "27960fcb-f56f-42e8-8233-430811c533e8" (UID: "27960fcb-f56f-42e8-8233-430811c533e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:35.147577 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.147547    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27960fcb-f56f-42e8-8233-430811c533e8-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.150123 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.150094    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27960fcb-f56f-42e8-8233-430811c533e8-kube-api-access-kc9vm" pod "27960fcb-f56f-42e8-8233-430811c533e8" (UID: "27960fcb-f56f-42e8-8233-430811c533e8"). InnerVolumeSpecName "kube-api-access-kc9vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:35.251359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.251295    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc9vm\" (UniqueName: \"kubernetes.io/projected/27960fcb-f56f-42e8-8233-430811c533e8-kube-api-access-kc9vm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:35.339415 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.339382    5893 generic.go:356] "Generic (PLEG): container finished" podID="27960fcb-f56f-42e8-8233-430811c533e8" containerID="14e70bc12788b902dd361909bd24ce61462975a653bd2a0078adf971f7c9d0ac" exitCode=0
Oct 30 13:20:35.339923 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.339431    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"
Oct 30 13:20:35.339923 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.339477    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw" event={"ID":"27960fcb-f56f-42e8-8233-430811c533e8","Type":"ContainerDied","Data":"14e70bc12788b902dd361909bd24ce61462975a653bd2a0078adf971f7c9d0ac"}
Oct 30 13:20:35.339923 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.339542    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw" event={"ID":"27960fcb-f56f-42e8-8233-430811c533e8","Type":"ContainerDied","Data":"df5b64a87de5ab0788550d2f2ece4e547989752f53ec4136a359e38335ee1f75"}
Oct 30 13:20:35.339923 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.339569    5893 scope.go:122] "RemoveContainer" containerID="14e70bc12788b902dd361909bd24ce61462975a653bd2a0078adf971f7c9d0ac"
Oct 30 13:20:35.354072 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.354052    5893 scope.go:122] "RemoveContainer" containerID="14e70bc12788b902dd361909bd24ce61462975a653bd2a0078adf971f7c9d0ac"
Oct 30 13:20:35.354422 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:20:35.354401    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14e70bc12788b902dd361909bd24ce61462975a653bd2a0078adf971f7c9d0ac\": container with ID starting with 14e70bc12788b902dd361909bd24ce61462975a653bd2a0078adf971f7c9d0ac not found: ID does not exist" containerID="14e70bc12788b902dd361909bd24ce61462975a653bd2a0078adf971f7c9d0ac"
Oct 30 13:20:35.354510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.354431    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14e70bc12788b902dd361909bd24ce61462975a653bd2a0078adf971f7c9d0ac"} err="failed to get container status \"14e70bc12788b902dd361909bd24ce61462975a653bd2a0078adf971f7c9d0ac\": rpc error: code = NotFound desc = could not find container \"14e70bc12788b902dd361909bd24ce61462975a653bd2a0078adf971f7c9d0ac\": container with ID starting with 14e70bc12788b902dd361909bd24ce61462975a653bd2a0078adf971f7c9d0ac not found: ID does not exist"
Oct 30 13:20:35.406884 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.406856    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"]
Oct 30 13:20:35.418097 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:35.418070    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-2a3e01-8b78d7988-mzbkw"]
Oct 30 13:20:36.534436 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:36.534408    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27960fcb-f56f-42e8-8233-430811c533e8" path="/var/lib/kubelet/pods/27960fcb-f56f-42e8-8233-430811c533e8/volumes"
Oct 30 13:20:38.378982 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:38.378940    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"]
Oct 30 13:20:38.379539 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:38.379213    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw" podUID="a2214ddd-9b56-4e64-8346-9f009202f5d4" containerName="istio-proxy" containerID="cri-o://8fc3e5cb0eb298b3e4945f9d134f8e769b5c9e01de5d03bf1f7505515e1ceded" gracePeriod=30
Oct 30 13:20:41.120445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:41.120261    5893 patch_prober.go:29] interesting pod/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:20:41.120445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:41.120346    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw" podUID="a2214ddd-9b56-4e64-8346-9f009202f5d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:20:42.021364 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:42.021332    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"]
Oct 30 13:20:42.054497 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:42.054461    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"]
Oct 30 13:20:43.668791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.668719    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:20:43.831694 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.831661    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2214ddd-9b56-4e64-8346-9f009202f5d4-kube-api-access-h7q42\" (UniqueName: \"kubernetes.io/projected/a2214ddd-9b56-4e64-8346-9f009202f5d4-kube-api-access-h7q42\") pod \"a2214ddd-9b56-4e64-8346-9f009202f5d4\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") "
Oct 30 13:20:43.831901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.831717    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-workload-socket\") pod \"a2214ddd-9b56-4e64-8346-9f009202f5d4\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") "
Oct 30 13:20:43.831901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.831850    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-data\") pod \"a2214ddd-9b56-4e64-8346-9f009202f5d4\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") "
Oct 30 13:20:43.831901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.831899    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-envoy\") pod \"a2214ddd-9b56-4e64-8346-9f009202f5d4\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") "
Oct 30 13:20:43.832091 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.832011    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-credential-socket\") pod \"a2214ddd-9b56-4e64-8346-9f009202f5d4\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") "
Oct 30 13:20:43.832091 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.832060    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-podinfo\") pod \"a2214ddd-9b56-4e64-8346-9f009202f5d4\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") "
Oct 30 13:20:43.832091 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.832045    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-workload-socket" pod "a2214ddd-9b56-4e64-8346-9f009202f5d4" (UID: "a2214ddd-9b56-4e64-8346-9f009202f5d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:43.832091 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.832079    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-token\" (UniqueName: \"kubernetes.io/projected/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-token\") pod \"a2214ddd-9b56-4e64-8346-9f009202f5d4\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") "
Oct 30 13:20:43.832293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.832111    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2214ddd-9b56-4e64-8346-9f009202f5d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2214ddd-9b56-4e64-8346-9f009202f5d4-istiod-ca-cert\") pod \"a2214ddd-9b56-4e64-8346-9f009202f5d4\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") "
Oct 30 13:20:43.832293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.832180    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-workload-certs\") pod \"a2214ddd-9b56-4e64-8346-9f009202f5d4\" (UID: \"a2214ddd-9b56-4e64-8346-9f009202f5d4\") "
Oct 30 13:20:43.832293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.832110    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-data" pod "a2214ddd-9b56-4e64-8346-9f009202f5d4" (UID: "a2214ddd-9b56-4e64-8346-9f009202f5d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:43.832467 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.832351    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-credential-socket" pod "a2214ddd-9b56-4e64-8346-9f009202f5d4" (UID: "a2214ddd-9b56-4e64-8346-9f009202f5d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:43.832526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.832502    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-workload-certs" pod "a2214ddd-9b56-4e64-8346-9f009202f5d4" (UID: "a2214ddd-9b56-4e64-8346-9f009202f5d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:43.832580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.832528    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2214ddd-9b56-4e64-8346-9f009202f5d4-istiod-ca-cert" pod "a2214ddd-9b56-4e64-8346-9f009202f5d4" (UID: "a2214ddd-9b56-4e64-8346-9f009202f5d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:43.833240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.833171    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.833240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.833201    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2214ddd-9b56-4e64-8346-9f009202f5d4-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.833240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.833215    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.833240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.833227    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.833240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.833241    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.834339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.834308    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-envoy" pod "a2214ddd-9b56-4e64-8346-9f009202f5d4" (UID: "a2214ddd-9b56-4e64-8346-9f009202f5d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:43.834432 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.834409    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-podinfo" pod "a2214ddd-9b56-4e64-8346-9f009202f5d4" (UID: "a2214ddd-9b56-4e64-8346-9f009202f5d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:20:43.834577 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.834558    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-token" pod "a2214ddd-9b56-4e64-8346-9f009202f5d4" (UID: "a2214ddd-9b56-4e64-8346-9f009202f5d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:43.834860 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.834845    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2214ddd-9b56-4e64-8346-9f009202f5d4-kube-api-access-h7q42" pod "a2214ddd-9b56-4e64-8346-9f009202f5d4" (UID: "a2214ddd-9b56-4e64-8346-9f009202f5d4"). InnerVolumeSpecName "kube-api-access-h7q42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:43.934856 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.934774    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.934856 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.934798    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.934856 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.934809    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7q42\" (UniqueName: \"kubernetes.io/projected/a2214ddd-9b56-4e64-8346-9f009202f5d4-kube-api-access-h7q42\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:43.934856 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:43.934818    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2214ddd-9b56-4e64-8346-9f009202f5d4-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:44.170962 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:44.170927    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"]
Oct 30 13:20:44.171262 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:44.171206    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" podUID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" containerName="istio-proxy" containerID="cri-o://fe7413e697a8f873b64c41e418c86ee426a4950be709f1aba33aeeb10952325a" gracePeriod=30
Oct 30 13:20:44.421760 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:44.421727    5893 generic.go:356] "Generic (PLEG): container finished" podID="a2214ddd-9b56-4e64-8346-9f009202f5d4" containerID="8fc3e5cb0eb298b3e4945f9d134f8e769b5c9e01de5d03bf1f7505515e1ceded" exitCode=0
Oct 30 13:20:44.421988 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:44.421783    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"
Oct 30 13:20:44.421988 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:44.421816    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw" event={"ID":"a2214ddd-9b56-4e64-8346-9f009202f5d4","Type":"ContainerDied","Data":"8fc3e5cb0eb298b3e4945f9d134f8e769b5c9e01de5d03bf1f7505515e1ceded"}
Oct 30 13:20:44.421988 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:44.421878    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw" event={"ID":"a2214ddd-9b56-4e64-8346-9f009202f5d4","Type":"ContainerDied","Data":"30bfda6f1bd86e49d4c76fb8fdb0f675a2a7b0ce3b4b74c57b860465d5ca1d00"}
Oct 30 13:20:44.421988 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:44.421905    5893 scope.go:122] "RemoveContainer" containerID="8fc3e5cb0eb298b3e4945f9d134f8e769b5c9e01de5d03bf1f7505515e1ceded"
Oct 30 13:20:44.433717 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:44.433699    5893 scope.go:122] "RemoveContainer" containerID="8fc3e5cb0eb298b3e4945f9d134f8e769b5c9e01de5d03bf1f7505515e1ceded"
Oct 30 13:20:44.434033 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:20:44.434011    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc3e5cb0eb298b3e4945f9d134f8e769b5c9e01de5d03bf1f7505515e1ceded\": container with ID starting with 8fc3e5cb0eb298b3e4945f9d134f8e769b5c9e01de5d03bf1f7505515e1ceded not found: ID does not exist" containerID="8fc3e5cb0eb298b3e4945f9d134f8e769b5c9e01de5d03bf1f7505515e1ceded"
Oct 30 13:20:44.434097 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:44.434048    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc3e5cb0eb298b3e4945f9d134f8e769b5c9e01de5d03bf1f7505515e1ceded"} err="failed to get container status \"8fc3e5cb0eb298b3e4945f9d134f8e769b5c9e01de5d03bf1f7505515e1ceded\": rpc error: code = NotFound desc = could not find container \"8fc3e5cb0eb298b3e4945f9d134f8e769b5c9e01de5d03bf1f7505515e1ceded\": container with ID starting with 8fc3e5cb0eb298b3e4945f9d134f8e769b5c9e01de5d03bf1f7505515e1ceded not found: ID does not exist"
Oct 30 13:20:44.499529 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:44.499488    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"]
Oct 30 13:20:44.503446 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:44.503418    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2a3e01-openshift-default-56dfc4f47-pvscw"]
Oct 30 13:20:44.534804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:44.534772    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2214ddd-9b56-4e64-8346-9f009202f5d4" path="/var/lib/kubelet/pods/a2214ddd-9b56-4e64-8346-9f009202f5d4/volumes"
Oct 30 13:20:45.343957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:45.343926    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"
Oct 30 13:20:47.029476 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:47.029436    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm" podUID="fca2cd17-7269-4207-be71-b2c5078a8e28" containerName="payload-processing" containerID="cri-o://74dc83cce5f91e1a97ed6139af7d9094171a6f3912cd4aedfc7779fca608920c" gracePeriod=25
Oct 30 13:20:47.055077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:47.055026    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr" podUID="d447e622-be3c-488d-b8ac-9a0713b0a643" containerName="payload-pre-processing" containerID="cri-o://a802146d310962a0e3e27a4a3226b279bbc7dc9ad0dbd643c1c362b7e10e0c78" gracePeriod=25
Oct 30 13:20:48.395983 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:48.395946    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-qg8wr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:20:48.396408 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:48.396002    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" podUID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:20:48.654681 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:20:48.654595    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:20:49.470142 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.470116    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 13:20:49.548941 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.548908    5893 generic.go:356] "Generic (PLEG): container finished" podID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" containerID="fe7413e697a8f873b64c41e418c86ee426a4950be709f1aba33aeeb10952325a" exitCode=0
Oct 30 13:20:49.549175 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.549156    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" event={"ID":"44ac65bd-b10a-477b-b196-9ffe6a902ca0","Type":"ContainerDied","Data":"fe7413e697a8f873b64c41e418c86ee426a4950be709f1aba33aeeb10952325a"}
Oct 30 13:20:49.549352 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.549335    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr" event={"ID":"44ac65bd-b10a-477b-b196-9ffe6a902ca0","Type":"ContainerDied","Data":"20e2bf9cbb3622f50c04028b150a9b92a886fcec86fc324cf24be9bbe1097163"}
Oct 30 13:20:49.549469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.549458    5893 scope.go:122] "RemoveContainer" containerID="fe7413e697a8f873b64c41e418c86ee426a4950be709f1aba33aeeb10952325a"
Oct 30 13:20:49.549719 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.549704    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"
Oct 30 13:20:49.565585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.565560    5893 scope.go:122] "RemoveContainer" containerID="fe7413e697a8f873b64c41e418c86ee426a4950be709f1aba33aeeb10952325a"
Oct 30 13:20:49.565945 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:20:49.565917    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe7413e697a8f873b64c41e418c86ee426a4950be709f1aba33aeeb10952325a\": container with ID starting with fe7413e697a8f873b64c41e418c86ee426a4950be709f1aba33aeeb10952325a not found: ID does not exist" containerID="fe7413e697a8f873b64c41e418c86ee426a4950be709f1aba33aeeb10952325a"
Oct 30 13:20:49.566027 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.565957    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe7413e697a8f873b64c41e418c86ee426a4950be709f1aba33aeeb10952325a"} err="failed to get container status \"fe7413e697a8f873b64c41e418c86ee426a4950be709f1aba33aeeb10952325a\": rpc error: code = NotFound desc = could not find container \"fe7413e697a8f873b64c41e418c86ee426a4950be709f1aba33aeeb10952325a\": container with ID starting with fe7413e697a8f873b64c41e418c86ee426a4950be709f1aba33aeeb10952325a not found: ID does not exist"
Oct 30 13:20:49.634555 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.634519    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ac65bd-b10a-477b-b196-9ffe6a902ca0-kube-api-access-k627m\" (UniqueName: \"kubernetes.io/projected/44ac65bd-b10a-477b-b196-9ffe6a902ca0-kube-api-access-k627m\") pod \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") "
Oct 30 13:20:49.634752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.634565    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-token\" (UniqueName: \"kubernetes.io/projected/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-token\") pod \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") "
Oct 30 13:20:49.634823 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.634788    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-data\") pod \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") "
Oct 30 13:20:49.635201 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.635162    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-data" pod "44ac65bd-b10a-477b-b196-9ffe6a902ca0" (UID: "44ac65bd-b10a-477b-b196-9ffe6a902ca0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:49.635978 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.635264    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-podinfo\") pod \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") "
Oct 30 13:20:49.636105 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.636045    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istiod-ca-cert\") pod \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") "
Oct 30 13:20:49.636105 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.636090    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-workload-socket\") pod \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") "
Oct 30 13:20:49.636238 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.636178    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-workload-certs\") pod \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") "
Oct 30 13:20:49.636771 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.636335    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-credential-socket\") pod \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") "
Oct 30 13:20:49.636771 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.636498    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-envoy\") pod \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\" (UID: \"44ac65bd-b10a-477b-b196-9ffe6a902ca0\") "
Oct 30 13:20:49.636930 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.636814    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-credential-socket" pod "44ac65bd-b10a-477b-b196-9ffe6a902ca0" (UID: "44ac65bd-b10a-477b-b196-9ffe6a902ca0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:49.638400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.637458    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-workload-certs" pod "44ac65bd-b10a-477b-b196-9ffe6a902ca0" (UID: "44ac65bd-b10a-477b-b196-9ffe6a902ca0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:49.638400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.638048    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-workload-socket" pod "44ac65bd-b10a-477b-b196-9ffe6a902ca0" (UID: "44ac65bd-b10a-477b-b196-9ffe6a902ca0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:49.638400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.638324    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:49.638400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.638346    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:49.638400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.638361    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:49.638400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.638374    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:49.639254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.639234    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istiod-ca-cert" pod "44ac65bd-b10a-477b-b196-9ffe6a902ca0" (UID: "44ac65bd-b10a-477b-b196-9ffe6a902ca0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:49.655689 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.655660    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-podinfo" pod "44ac65bd-b10a-477b-b196-9ffe6a902ca0" (UID: "44ac65bd-b10a-477b-b196-9ffe6a902ca0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:20:49.655996 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.655863    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-token" pod "44ac65bd-b10a-477b-b196-9ffe6a902ca0" (UID: "44ac65bd-b10a-477b-b196-9ffe6a902ca0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:49.657749 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.657729    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-envoy" pod "44ac65bd-b10a-477b-b196-9ffe6a902ca0" (UID: "44ac65bd-b10a-477b-b196-9ffe6a902ca0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:20:49.657837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.657816    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ac65bd-b10a-477b-b196-9ffe6a902ca0-kube-api-access-k627m" pod "44ac65bd-b10a-477b-b196-9ffe6a902ca0" (UID: "44ac65bd-b10a-477b-b196-9ffe6a902ca0"). InnerVolumeSpecName "kube-api-access-k627m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:49.739826 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.739664    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:49.739826 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.739699    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k627m\" (UniqueName: \"kubernetes.io/projected/44ac65bd-b10a-477b-b196-9ffe6a902ca0-kube-api-access-k627m\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:49.739826 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.739713    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:49.739826 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.739727    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:49.739826 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.739739    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44ac65bd-b10a-477b-b196-9ffe6a902ca0-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:20:49.918897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.918866    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"]
Oct 30 13:20:49.923636 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:49.923609    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-qg8wr"]
Oct 30 13:20:50.533819 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:50.533783    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" path="/var/lib/kubelet/pods/44ac65bd-b10a-477b-b196-9ffe6a902ca0/volumes"
Oct 30 13:20:54.981146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:54.981110    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr" podUID="d447e622-be3c-488d-b8ac-9a0713b0a643" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:20:55.988451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:20:55.988417    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm" podUID="fca2cd17-7269-4207-be71-b2c5078a8e28" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:21:01.052628 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:01.052600    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:21:01.082433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:01.082405    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:21:01.083423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:01.083395    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:21:01.096648 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:01.096623    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:21:04.802764 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:04.802718    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"]
Oct 30 13:21:04.803243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:04.802944    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f" podUID="82b8dfd6-499a-4167-9718-cd5fa7a5c055" containerName="discovery" containerID="cri-o://7172f195e6ca62a7d78cf3913d1be62756ebb6508ac06036a4f238adc1f9ce97" gracePeriod=30
Oct 30 13:21:04.981498 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:04.981457    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr" podUID="d447e622-be3c-488d-b8ac-9a0713b0a643" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:21:05.112948 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.112922    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:21:05.213616 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.213573    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-kubeconfig\") pod \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") "
Oct 30 13:21:05.213788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.213656    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-csr-dns-cert\") pod \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") "
Oct 30 13:21:05.213788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.213707    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b8dfd6-499a-4167-9718-cd5fa7a5c055-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b8dfd6-499a-4167-9718-cd5fa7a5c055-local-certs\") pod \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") "
Oct 30 13:21:05.214224 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.213903    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-csr-ca-configmap\") pod \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") "
Oct 30 13:21:05.214224 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.214051    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-cacerts\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-cacerts\") pod \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") "
Oct 30 13:21:05.214224 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.214199    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-token\" (UniqueName: \"kubernetes.io/projected/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-token\") pod \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") "
Oct 30 13:21:05.214503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.214254    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b8dfd6-499a-4167-9718-cd5fa7a5c055-kube-api-access-xr2t6\" (UniqueName: \"kubernetes.io/projected/82b8dfd6-499a-4167-9718-cd5fa7a5c055-kube-api-access-xr2t6\") pod \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\" (UID: \"82b8dfd6-499a-4167-9718-cd5fa7a5c055\") "
Oct 30 13:21:05.214503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.214386    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-csr-ca-configmap" pod "82b8dfd6-499a-4167-9718-cd5fa7a5c055" (UID: "82b8dfd6-499a-4167-9718-cd5fa7a5c055"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:05.214923 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.214901    5893 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-csr-ca-configmap\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:05.217965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.217938    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b8dfd6-499a-4167-9718-cd5fa7a5c055-kube-api-access-xr2t6" pod "82b8dfd6-499a-4167-9718-cd5fa7a5c055" (UID: "82b8dfd6-499a-4167-9718-cd5fa7a5c055"). InnerVolumeSpecName "kube-api-access-xr2t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:05.219093 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.219066    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-token" pod "82b8dfd6-499a-4167-9718-cd5fa7a5c055" (UID: "82b8dfd6-499a-4167-9718-cd5fa7a5c055"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:05.223098 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.221698    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-kubeconfig" pod "82b8dfd6-499a-4167-9718-cd5fa7a5c055" (UID: "82b8dfd6-499a-4167-9718-cd5fa7a5c055"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:05.223098 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.221697    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b8dfd6-499a-4167-9718-cd5fa7a5c055-local-certs" pod "82b8dfd6-499a-4167-9718-cd5fa7a5c055" (UID: "82b8dfd6-499a-4167-9718-cd5fa7a5c055"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:05.223098 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.221910    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-csr-dns-cert" pod "82b8dfd6-499a-4167-9718-cd5fa7a5c055" (UID: "82b8dfd6-499a-4167-9718-cd5fa7a5c055"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:05.223421 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.223371    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-cacerts" pod "82b8dfd6-499a-4167-9718-cd5fa7a5c055" (UID: "82b8dfd6-499a-4167-9718-cd5fa7a5c055"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:05.316132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.316092    5893 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-cacerts\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:05.316132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.316119    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:05.316132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.316131    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr2t6\" (UniqueName: \"kubernetes.io/projected/82b8dfd6-499a-4167-9718-cd5fa7a5c055-kube-api-access-xr2t6\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:05.316132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.316140    5893 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-kubeconfig\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:05.316498 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.316149    5893 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/82b8dfd6-499a-4167-9718-cd5fa7a5c055-istio-csr-dns-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:05.316498 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.316158    5893 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b8dfd6-499a-4167-9718-cd5fa7a5c055-local-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:05.711771 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.711731    5893 generic.go:356] "Generic (PLEG): container finished" podID="82b8dfd6-499a-4167-9718-cd5fa7a5c055" containerID="7172f195e6ca62a7d78cf3913d1be62756ebb6508ac06036a4f238adc1f9ce97" exitCode=0
Oct 30 13:21:05.711957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.711783    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"
Oct 30 13:21:05.711957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.711818    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f" event={"ID":"82b8dfd6-499a-4167-9718-cd5fa7a5c055","Type":"ContainerDied","Data":"7172f195e6ca62a7d78cf3913d1be62756ebb6508ac06036a4f238adc1f9ce97"}
Oct 30 13:21:05.711957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.711889    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f" event={"ID":"82b8dfd6-499a-4167-9718-cd5fa7a5c055","Type":"ContainerDied","Data":"b43c14cb6f6bdad788f0ba3954d000d52aa64f78c7209f2e73cab5931d35ef20"}
Oct 30 13:21:05.711957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.711915    5893 scope.go:122] "RemoveContainer" containerID="7172f195e6ca62a7d78cf3913d1be62756ebb6508ac06036a4f238adc1f9ce97"
Oct 30 13:21:05.724167 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.724143    5893 scope.go:122] "RemoveContainer" containerID="7172f195e6ca62a7d78cf3913d1be62756ebb6508ac06036a4f238adc1f9ce97"
Oct 30 13:21:05.724466 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:21:05.724446    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7172f195e6ca62a7d78cf3913d1be62756ebb6508ac06036a4f238adc1f9ce97\": container with ID starting with 7172f195e6ca62a7d78cf3913d1be62756ebb6508ac06036a4f238adc1f9ce97 not found: ID does not exist" containerID="7172f195e6ca62a7d78cf3913d1be62756ebb6508ac06036a4f238adc1f9ce97"
Oct 30 13:21:05.724532 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.724475    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7172f195e6ca62a7d78cf3913d1be62756ebb6508ac06036a4f238adc1f9ce97"} err="failed to get container status \"7172f195e6ca62a7d78cf3913d1be62756ebb6508ac06036a4f238adc1f9ce97\": rpc error: code = NotFound desc = could not find container \"7172f195e6ca62a7d78cf3913d1be62756ebb6508ac06036a4f238adc1f9ce97\": container with ID starting with 7172f195e6ca62a7d78cf3913d1be62756ebb6508ac06036a4f238adc1f9ce97 not found: ID does not exist"
Oct 30 13:21:05.800387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.800348    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"]
Oct 30 13:21:05.801892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.801859    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-q8n9f"]
Oct 30 13:21:05.988515 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:05.988480    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm" podUID="fca2cd17-7269-4207-be71-b2c5078a8e28" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:21:06.533367 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:06.533334    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b8dfd6-499a-4167-9718-cd5fa7a5c055" path="/var/lib/kubelet/pods/82b8dfd6-499a-4167-9718-cd5fa7a5c055/volumes"
Oct 30 13:21:07.256872 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.256848    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"
Oct 30 13:21:07.346848 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.346799    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fca2cd17-7269-4207-be71-b2c5078a8e28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca2cd17-7269-4207-be71-b2c5078a8e28-plugins-config-volume\") pod \"fca2cd17-7269-4207-be71-b2c5078a8e28\" (UID: \"fca2cd17-7269-4207-be71-b2c5078a8e28\") "
Oct 30 13:21:07.347005 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.346866    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fca2cd17-7269-4207-be71-b2c5078a8e28-kube-api-access-q8z96\" (UniqueName: \"kubernetes.io/projected/fca2cd17-7269-4207-be71-b2c5078a8e28-kube-api-access-q8z96\") pod \"fca2cd17-7269-4207-be71-b2c5078a8e28\" (UID: \"fca2cd17-7269-4207-be71-b2c5078a8e28\") "
Oct 30 13:21:07.347256 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.347228    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fca2cd17-7269-4207-be71-b2c5078a8e28-plugins-config-volume" pod "fca2cd17-7269-4207-be71-b2c5078a8e28" (UID: "fca2cd17-7269-4207-be71-b2c5078a8e28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:07.347703 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.347683    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fca2cd17-7269-4207-be71-b2c5078a8e28-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:07.349623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.349595    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fca2cd17-7269-4207-be71-b2c5078a8e28-kube-api-access-q8z96" pod "fca2cd17-7269-4207-be71-b2c5078a8e28" (UID: "fca2cd17-7269-4207-be71-b2c5078a8e28"). InnerVolumeSpecName "kube-api-access-q8z96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:07.365875 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.365848    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"
Oct 30 13:21:07.454085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.454044    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d447e622-be3c-488d-b8ac-9a0713b0a643-kube-api-access-wj66x\" (UniqueName: \"kubernetes.io/projected/d447e622-be3c-488d-b8ac-9a0713b0a643-kube-api-access-wj66x\") pod \"d447e622-be3c-488d-b8ac-9a0713b0a643\" (UID: \"d447e622-be3c-488d-b8ac-9a0713b0a643\") "
Oct 30 13:21:07.454425 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.454391    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d447e622-be3c-488d-b8ac-9a0713b0a643-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d447e622-be3c-488d-b8ac-9a0713b0a643-plugins-config-volume\") pod \"d447e622-be3c-488d-b8ac-9a0713b0a643\" (UID: \"d447e622-be3c-488d-b8ac-9a0713b0a643\") "
Oct 30 13:21:07.455498 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.455464    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d447e622-be3c-488d-b8ac-9a0713b0a643-plugins-config-volume" pod "d447e622-be3c-488d-b8ac-9a0713b0a643" (UID: "d447e622-be3c-488d-b8ac-9a0713b0a643"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:07.456094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.456064    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8z96\" (UniqueName: \"kubernetes.io/projected/fca2cd17-7269-4207-be71-b2c5078a8e28-kube-api-access-q8z96\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:07.456228 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.456096    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d447e622-be3c-488d-b8ac-9a0713b0a643-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:07.458390 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.458361    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d447e622-be3c-488d-b8ac-9a0713b0a643-kube-api-access-wj66x" pod "d447e622-be3c-488d-b8ac-9a0713b0a643" (UID: "d447e622-be3c-488d-b8ac-9a0713b0a643"). InnerVolumeSpecName "kube-api-access-wj66x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:07.557681 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.557648    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj66x\" (UniqueName: \"kubernetes.io/projected/d447e622-be3c-488d-b8ac-9a0713b0a643-kube-api-access-wj66x\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:07.730607 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.730572    5893 generic.go:356] "Generic (PLEG): container finished" podID="d447e622-be3c-488d-b8ac-9a0713b0a643" containerID="a802146d310962a0e3e27a4a3226b279bbc7dc9ad0dbd643c1c362b7e10e0c78" exitCode=0
Oct 30 13:21:07.730793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.730629    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"
Oct 30 13:21:07.730793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.730652    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr" event={"ID":"d447e622-be3c-488d-b8ac-9a0713b0a643","Type":"ContainerDied","Data":"a802146d310962a0e3e27a4a3226b279bbc7dc9ad0dbd643c1c362b7e10e0c78"}
Oct 30 13:21:07.730793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.730705    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr" event={"ID":"d447e622-be3c-488d-b8ac-9a0713b0a643","Type":"ContainerDied","Data":"631f8b88ca781dbb2981212f632ba538af41118afe34db60a561dfe622e7bce7"}
Oct 30 13:21:07.730793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.730727    5893 scope.go:122] "RemoveContainer" containerID="a802146d310962a0e3e27a4a3226b279bbc7dc9ad0dbd643c1c362b7e10e0c78"
Oct 30 13:21:07.732611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.732586    5893 generic.go:356] "Generic (PLEG): container finished" podID="fca2cd17-7269-4207-be71-b2c5078a8e28" containerID="74dc83cce5f91e1a97ed6139af7d9094171a6f3912cd4aedfc7779fca608920c" exitCode=0
Oct 30 13:21:07.732762 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.732642    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"
Oct 30 13:21:07.732762 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.732727    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm" event={"ID":"fca2cd17-7269-4207-be71-b2c5078a8e28","Type":"ContainerDied","Data":"74dc83cce5f91e1a97ed6139af7d9094171a6f3912cd4aedfc7779fca608920c"}
Oct 30 13:21:07.732871 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.732777    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm" event={"ID":"fca2cd17-7269-4207-be71-b2c5078a8e28","Type":"ContainerDied","Data":"92b1db049588a6a9c5b9485f8cba74f696d40b690d70bb3ed92bb962441a4f8a"}
Oct 30 13:21:07.744571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.744551    5893 scope.go:122] "RemoveContainer" containerID="a802146d310962a0e3e27a4a3226b279bbc7dc9ad0dbd643c1c362b7e10e0c78"
Oct 30 13:21:07.744918 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:21:07.744892    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a802146d310962a0e3e27a4a3226b279bbc7dc9ad0dbd643c1c362b7e10e0c78\": container with ID starting with a802146d310962a0e3e27a4a3226b279bbc7dc9ad0dbd643c1c362b7e10e0c78 not found: ID does not exist" containerID="a802146d310962a0e3e27a4a3226b279bbc7dc9ad0dbd643c1c362b7e10e0c78"
Oct 30 13:21:07.745029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.744930    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a802146d310962a0e3e27a4a3226b279bbc7dc9ad0dbd643c1c362b7e10e0c78"} err="failed to get container status \"a802146d310962a0e3e27a4a3226b279bbc7dc9ad0dbd643c1c362b7e10e0c78\": rpc error: code = NotFound desc = could not find container \"a802146d310962a0e3e27a4a3226b279bbc7dc9ad0dbd643c1c362b7e10e0c78\": container with ID starting with a802146d310962a0e3e27a4a3226b279bbc7dc9ad0dbd643c1c362b7e10e0c78 not found: ID does not exist"
Oct 30 13:21:07.745029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.744953    5893 scope.go:122] "RemoveContainer" containerID="74dc83cce5f91e1a97ed6139af7d9094171a6f3912cd4aedfc7779fca608920c"
Oct 30 13:21:07.755330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.755305    5893 scope.go:122] "RemoveContainer" containerID="74dc83cce5f91e1a97ed6139af7d9094171a6f3912cd4aedfc7779fca608920c"
Oct 30 13:21:07.755648 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:21:07.755622    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74dc83cce5f91e1a97ed6139af7d9094171a6f3912cd4aedfc7779fca608920c\": container with ID starting with 74dc83cce5f91e1a97ed6139af7d9094171a6f3912cd4aedfc7779fca608920c not found: ID does not exist" containerID="74dc83cce5f91e1a97ed6139af7d9094171a6f3912cd4aedfc7779fca608920c"
Oct 30 13:21:07.755747 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.755656    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74dc83cce5f91e1a97ed6139af7d9094171a6f3912cd4aedfc7779fca608920c"} err="failed to get container status \"74dc83cce5f91e1a97ed6139af7d9094171a6f3912cd4aedfc7779fca608920c\": rpc error: code = NotFound desc = could not find container \"74dc83cce5f91e1a97ed6139af7d9094171a6f3912cd4aedfc7779fca608920c\": container with ID starting with 74dc83cce5f91e1a97ed6139af7d9094171a6f3912cd4aedfc7779fca608920c not found: ID does not exist"
Oct 30 13:21:07.785166 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.785137    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"]
Oct 30 13:21:07.798516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.798489    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4a8ee1-54dfc45bc6-nqmmm"]
Oct 30 13:21:07.822708 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.822686    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"]
Oct 30 13:21:07.830156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:07.830131    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4a8ee1-576bdb4db7-67tpr"]
Oct 30 13:21:08.537354 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:08.537323    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d447e622-be3c-488d-b8ac-9a0713b0a643" path="/var/lib/kubelet/pods/d447e622-be3c-488d-b8ac-9a0713b0a643/volumes"
Oct 30 13:21:08.537866 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:08.537851    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fca2cd17-7269-4207-be71-b2c5078a8e28" path="/var/lib/kubelet/pods/fca2cd17-7269-4207-be71-b2c5078a8e28/volumes"
Oct 30 13:21:09.063822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.063681    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"]
Oct 30 13:21:09.064594 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064575    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d447e622-be3c-488d-b8ac-9a0713b0a643" containerName="payload-pre-processing"
Oct 30 13:21:09.064682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064596    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d447e622-be3c-488d-b8ac-9a0713b0a643" containerName="payload-pre-processing"
Oct 30 13:21:09.064682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064610    5893 cpu_manager.go:414] "containerMap: removing container" podUID="27960fcb-f56f-42e8-8233-430811c533e8" containerName="payload-processing"
Oct 30 13:21:09.064682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064616    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27960fcb-f56f-42e8-8233-430811c533e8" containerName="payload-processing"
Oct 30 13:21:09.064682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064640    5893 cpu_manager.go:414] "containerMap: removing container" podUID="a2214ddd-9b56-4e64-8346-9f009202f5d4" containerName="istio-proxy"
Oct 30 13:21:09.064682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064644    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2214ddd-9b56-4e64-8346-9f009202f5d4" containerName="istio-proxy"
Oct 30 13:21:09.064682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064654    5893 cpu_manager.go:414] "containerMap: removing container" podUID="fca2cd17-7269-4207-be71-b2c5078a8e28" containerName="payload-processing"
Oct 30 13:21:09.064682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064659    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fca2cd17-7269-4207-be71-b2c5078a8e28" containerName="payload-processing"
Oct 30 13:21:09.064896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064698    5893 cpu_manager.go:414] "containerMap: removing container" podUID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" containerName="istio-proxy"
Oct 30 13:21:09.064896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064703    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" containerName="istio-proxy"
Oct 30 13:21:09.064896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064712    5893 cpu_manager.go:414] "containerMap: removing container" podUID="82b8dfd6-499a-4167-9718-cd5fa7a5c055" containerName="discovery"
Oct 30 13:21:09.064896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064716    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b8dfd6-499a-4167-9718-cd5fa7a5c055" containerName="discovery"
Oct 30 13:21:09.064896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064811    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="44ac65bd-b10a-477b-b196-9ffe6a902ca0" containerName="istio-proxy"
Oct 30 13:21:09.064896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064819    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b8dfd6-499a-4167-9718-cd5fa7a5c055" containerName="discovery"
Oct 30 13:21:09.064896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064826    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d447e622-be3c-488d-b8ac-9a0713b0a643" containerName="payload-pre-processing"
Oct 30 13:21:09.064896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064834    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="27960fcb-f56f-42e8-8233-430811c533e8" containerName="payload-processing"
Oct 30 13:21:09.064896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064843    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2214ddd-9b56-4e64-8346-9f009202f5d4" containerName="istio-proxy"
Oct 30 13:21:09.064896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.064850    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="fca2cd17-7269-4207-be71-b2c5078a8e28" containerName="payload-processing"
Oct 30 13:21:09.070142 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.070122    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.091457 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.091434    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"]
Oct 30 13:21:09.099301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.099259    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bec983-openshift-default-dockercfg-4ktqh\""
Oct 30 13:21:09.181712 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.181685    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-workload-socket\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.181886 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.181722    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-credential-socket\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.181886 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.181754    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c179d0-e114-4d43-b296-db22061f87d6-istio-podinfo\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.181886 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.181842    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-istio-data\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.181996 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.181889    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c179d0-e114-4d43-b296-db22061f87d6-istio-token\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.182082 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.182066    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28z8w\" (UniqueName: \"kubernetes.io/projected/09c179d0-e114-4d43-b296-db22061f87d6-kube-api-access-28z8w\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.182118 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.182099    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c179d0-e114-4d43-b296-db22061f87d6-istiod-ca-cert\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.182233 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.182211    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-istio-envoy\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.182325 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.182312    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-workload-certs\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.283316 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.283260    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c179d0-e114-4d43-b296-db22061f87d6-istiod-ca-cert\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.283484 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.283367    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-istio-envoy\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.283484 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.283434    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-workload-certs\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.283630 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.283503    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-workload-socket\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.283630 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.283538    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-credential-socket\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.283630 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.283584    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c179d0-e114-4d43-b296-db22061f87d6-istio-podinfo\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.283630 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.283613    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-istio-data\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.283822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.283645    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c179d0-e114-4d43-b296-db22061f87d6-istio-token\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.283822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.283746    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-workload-certs\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.283932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.283847    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-workload-socket\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.283932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.283922    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28z8w\" (UniqueName: \"kubernetes.io/projected/09c179d0-e114-4d43-b296-db22061f87d6-kube-api-access-28z8w\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.284027 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.283994    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-credential-socket\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.284085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.284020    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c179d0-e114-4d43-b296-db22061f87d6-istiod-ca-cert\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.284210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.284191    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-istio-data\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.286518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.286497    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-istio-envoy\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.286659 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.286643    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c179d0-e114-4d43-b296-db22061f87d6-istio-podinfo\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.307199 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.307164    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c179d0-e114-4d43-b296-db22061f87d6-istio-token\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.307419 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.307394    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28z8w\" (UniqueName: \"kubernetes.io/projected/09c179d0-e114-4d43-b296-db22061f87d6-kube-api-access-28z8w\") pod \"e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") " pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.384748 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.384655    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:09.445209 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.445178    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"]
Oct 30 13:21:09.460394 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.460372    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.472961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.472935    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"]
Oct 30 13:21:09.477342 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.476724    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-618c72\"/\"kube-root-ca.crt\""
Oct 30 13:21:09.477342 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.476793    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-618c72\"/\"openshift-service-ca.crt\""
Oct 30 13:21:09.477342 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.476882    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-618c72\"/\"test-model-618c72-kserve-self-signed-certs\""
Oct 30 13:21:09.477342 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.476969    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-618c72\"/\"default-dockercfg-kmmsm\""
Oct 30 13:21:09.588431 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.588391    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4356b5cd-aabc-41d4-89aa-d50586ce1952-tls-certs\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.588898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.588523    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-dshm\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.588898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.588629    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-model-cache\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.588898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.588780    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-home\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.588898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.588849    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-tmp-dir\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.589074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.588959    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgm6c\" (UniqueName: \"kubernetes.io/projected/4356b5cd-aabc-41d4-89aa-d50586ce1952-kube-api-access-wgm6c\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.690912 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.690811    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-home\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.691078 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.690932    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-tmp-dir\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.691473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.691444    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgm6c\" (UniqueName: \"kubernetes.io/projected/4356b5cd-aabc-41d4-89aa-d50586ce1952-kube-api-access-wgm6c\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.691936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.691911    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4356b5cd-aabc-41d4-89aa-d50586ce1952-tls-certs\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.692036 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.691986    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-dshm\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.692102 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.692083    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-model-cache\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.693500 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.693471    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-home\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.694803 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.694776    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-tmp-dir\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.695072 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.695023    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-model-cache\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.696574 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.696544    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-dshm\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.697463 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.697432    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4356b5cd-aabc-41d4-89aa-d50586ce1952-tls-certs\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.718265 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.718239    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgm6c\" (UniqueName: \"kubernetes.io/projected/4356b5cd-aabc-41d4-89aa-d50586ce1952-kube-api-access-wgm6c\") pod \"test-model-618c72-kserve-bb59f496f-wcxjm\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") " pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.771686 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.733487    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"]
Oct 30 13:21:09.804298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.797606    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:09.824328 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.821703    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:21:09.824328 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.821789    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:21:09.824328 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:09.821825    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:21:10.147249 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:21:10.146483    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4356b5cd_aabc_41d4_89aa_d50586ce1952.slice/crio-c6573e69d0e14195b7cc66322583e79cf94c9ee4e1e8b892853e614fdf1aa2b4 WatchSource:0}: Error finding container c6573e69d0e14195b7cc66322583e79cf94c9ee4e1e8b892853e614fdf1aa2b4: Status 404 returned error can't find the container with id c6573e69d0e14195b7cc66322583e79cf94c9ee4e1e8b892853e614fdf1aa2b4
Oct 30 13:21:10.147249 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:10.146781    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"]
Oct 30 13:21:10.835053 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:10.835019    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv" event={"ID":"09c179d0-e114-4d43-b296-db22061f87d6","Type":"ContainerStarted","Data":"78e8e3b484080071adffbe3ecb1874c595ba6887626ac219ea2c7be910543478"}
Oct 30 13:21:10.835534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:10.835076    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv" event={"ID":"09c179d0-e114-4d43-b296-db22061f87d6","Type":"ContainerStarted","Data":"eddd21726c2ec82efb4f76b539fe98750729ed4c36e302e35fd5391506f7c966"}
Oct 30 13:21:10.837088 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:10.837052    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm" event={"ID":"4356b5cd-aabc-41d4-89aa-d50586ce1952","Type":"ContainerStarted","Data":"19bb7f26dcff56674d0f36edcb016bcdaa08d67fb7a599bc99d67b386944b76a"}
Oct 30 13:21:10.837223 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:10.837098    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm" event={"ID":"4356b5cd-aabc-41d4-89aa-d50586ce1952","Type":"ContainerStarted","Data":"c6573e69d0e14195b7cc66322583e79cf94c9ee4e1e8b892853e614fdf1aa2b4"}
Oct 30 13:21:10.960303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:10.950100    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm" podStartSLOduration=1.950078779 podStartE2EDuration="1.950078779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:10.948934229 +0000 UTC m=+2112.532363924" watchObservedRunningTime="2026-10-30 13:21:10.950078779 +0000 UTC m=+2112.533508473"
Oct 30 13:21:10.960303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:10.950709    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv" podStartSLOduration=1.95069801 podStartE2EDuration="1.95069801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:10.899338854 +0000 UTC m=+2112.482768564" watchObservedRunningTime="2026-10-30 13:21:10.95069801 +0000 UTC m=+2112.534127705"
Oct 30 13:21:11.385377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:11.385324    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:11.387415 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:11.387392    5893 patch_prober.go:29] interesting pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:21:11.387528 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:11.387436    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv" podUID="09c179d0-e114-4d43-b296-db22061f87d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 13:21:11.801770 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:11.801732    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:12.385952 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:12.385917    5893 patch_prober.go:29] interesting pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:21:12.386347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:12.385988    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv" podUID="09c179d0-e114-4d43-b296-db22061f87d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 13:21:13.385331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:13.385249    5893 patch_prober.go:29] interesting pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:21:13.385606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:13.385349    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv" podUID="09c179d0-e114-4d43-b296-db22061f87d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 13:21:14.386698 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:14.386664    5893 patch_prober.go:29] interesting pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:21:14.387154 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:14.386727    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv" podUID="09c179d0-e114-4d43-b296-db22061f87d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 13:21:15.385806 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:15.385760    5893 patch_prober.go:29] interesting pod/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body=
Oct 30 13:21:15.386029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:15.385829    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv" podUID="09c179d0-e114-4d43-b296-db22061f87d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused"
Oct 30 13:21:16.346477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.346441    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"]
Oct 30 13:21:16.353075 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.353035    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.387037 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.387009    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-80fbd9-openshift-default-dockercfg-c2r4s\""
Oct 30 13:21:16.393812 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.393776    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"]
Oct 30 13:21:16.426995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.426961    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-workload-socket\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.427258 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.427234    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-token\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.427449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.427432    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-envoy\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.427595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.427580    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379ef92e-3861-4eec-8eb2-c52787142d1a-istiod-ca-cert\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.427739 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.427721    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26pb4\" (UniqueName: \"kubernetes.io/projected/379ef92e-3861-4eec-8eb2-c52787142d1a-kube-api-access-26pb4\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.427976 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.427956    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-credential-socket\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.428177 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.428158    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-workload-certs\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.428342 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.428324    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-podinfo\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.428478 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.428463    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-data\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.530649 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.530611    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-credential-socket\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.530855 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.530700    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-workload-certs\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.530855 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.530730    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-podinfo\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.530855 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.530757    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-data\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.530855 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.530846    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-workload-socket\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.531109 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.530881    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-token\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.531109 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.530914    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-envoy\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.531109 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.530977    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379ef92e-3861-4eec-8eb2-c52787142d1a-istiod-ca-cert\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.531109 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.531030    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26pb4\" (UniqueName: \"kubernetes.io/projected/379ef92e-3861-4eec-8eb2-c52787142d1a-kube-api-access-26pb4\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.531109 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.531064    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-credential-socket\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.531427 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.531405    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-data\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.531478 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.531439    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-workload-certs\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.531514 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.531468    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-workload-socket\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.532033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.532008    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379ef92e-3861-4eec-8eb2-c52787142d1a-istiod-ca-cert\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.533412 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.533391    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-podinfo\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.534022 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.533996    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-envoy\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.536055 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.536040    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:16.536431 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.536411    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:16.537173 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.537157    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:21:16.555023 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.554992    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-token\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.557343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.557319    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26pb4\" (UniqueName: \"kubernetes.io/projected/379ef92e-3861-4eec-8eb2-c52787142d1a-kube-api-access-26pb4\") pod \"e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") " pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:16.691186 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:16.674387    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:17.023424 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:17.023399    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"]
Oct 30 13:21:17.024664 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:21:17.024639    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod379ef92e_3861_4eec_8eb2_c52787142d1a.slice/crio-8536e1bba7a929dd31f0e81e5d2fd311f9703114c1882a2c42e06e393dec96b5 WatchSource:0}: Error finding container 8536e1bba7a929dd31f0e81e5d2fd311f9703114c1882a2c42e06e393dec96b5: Status 404 returned error can't find the container with id 8536e1bba7a929dd31f0e81e5d2fd311f9703114c1882a2c42e06e393dec96b5
Oct 30 13:21:17.027702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:17.027669    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:21:17.027837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:17.027773    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:21:17.027837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:17.027823    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:21:17.163298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:17.154423    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"]
Oct 30 13:21:17.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:17.984121    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5" event={"ID":"379ef92e-3861-4eec-8eb2-c52787142d1a","Type":"ContainerStarted","Data":"3705a4f895d3d183b469bc76933be5bc4ecfbd490714362263411807ab8a0d6e"}
Oct 30 13:21:17.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:17.984176    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5" event={"ID":"379ef92e-3861-4eec-8eb2-c52787142d1a","Type":"ContainerStarted","Data":"8536e1bba7a929dd31f0e81e5d2fd311f9703114c1882a2c42e06e393dec96b5"}
Oct 30 13:21:18.675381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:18.675342    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:18.677323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:18.677265    5893 patch_prober.go:29] interesting pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:21:18.677471 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:18.677346    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5" podUID="379ef92e-3861-4eec-8eb2-c52787142d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 13:21:18.679311 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:21:18.679265    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:21:19.679004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:19.678966    5893 patch_prober.go:29] interesting pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:21:19.679635 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:19.679604    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5" podUID="379ef92e-3861-4eec-8eb2-c52787142d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 13:21:19.801570 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:19.801540    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:19.814657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:19.814631    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:19.873298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:19.873170    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5" podStartSLOduration=3.873146597 podStartE2EDuration="3.873146597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:18.032631774 +0000 UTC m=+2119.616061479" watchObservedRunningTime="2026-10-30 13:21:19.873146597 +0000 UTC m=+2121.456576291"
Oct 30 13:21:20.013634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:20.013603    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:21:20.677435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:20.675990    5893 patch_prober.go:29] interesting pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:21:20.677435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:20.676048    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5" podUID="379ef92e-3861-4eec-8eb2-c52787142d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 13:21:21.675464 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:21.675382    5893 patch_prober.go:29] interesting pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:21:21.675464 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:21.675443    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5" podUID="379ef92e-3861-4eec-8eb2-c52787142d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 13:21:22.157437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:22.157351    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz" podUID="dc27afd0-973b-4e86-9ce1-adc6d294d7be" containerName="payload-processing" containerID="cri-o://6a8a9507b0b5a4982d342fa59db603305d89065bd37279d6637b1893c5c2d1cd" gracePeriod=25
Oct 30 13:21:22.675793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:22.675754    5893 patch_prober.go:29] interesting pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:21:22.676337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:22.676264    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5" podUID="379ef92e-3861-4eec-8eb2-c52787142d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 13:21:23.675514 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:23.675476    5893 patch_prober.go:29] interesting pod/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Oct 30 13:21:23.675708 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:23.675545    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5" podUID="379ef92e-3861-4eec-8eb2-c52787142d1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Oct 30 13:21:24.683000 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:24.682964    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:24.683784 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:24.683487    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:24.684611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:24.684586    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:21:29.082046 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:29.082005    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz" podUID="dc27afd0-973b-4e86-9ce1-adc6d294d7be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:21:39.081739 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:39.081704    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz" podUID="dc27afd0-973b-4e86-9ce1-adc6d294d7be" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:21:41.247798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.247758    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"]
Oct 30 13:21:41.248199 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.248011    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6" podUID="62c80976-1d79-48e5-8718-2695ae37d011" containerName="maas-api" containerID="cri-o://365d2a330d543c11bd861b6c03ff496a23a92c322e10e9711a085c2f398c438f" gracePeriod=30
Oct 30 13:21:41.649546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.649503    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:21:41.857841 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.846346    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62c80976-1d79-48e5-8718-2695ae37d011-metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c80976-1d79-48e5-8718-2695ae37d011-metrics-certs\") pod \"62c80976-1d79-48e5-8718-2695ae37d011\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") "
Oct 30 13:21:41.858187 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.858164    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62c80976-1d79-48e5-8718-2695ae37d011-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62c80976-1d79-48e5-8718-2695ae37d011-maas-api-tls\") pod \"62c80976-1d79-48e5-8718-2695ae37d011\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") "
Oct 30 13:21:41.858508 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.858477    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c80976-1d79-48e5-8718-2695ae37d011-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c80976-1d79-48e5-8718-2695ae37d011-openshift-service-ca\") pod \"62c80976-1d79-48e5-8718-2695ae37d011\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") "
Oct 30 13:21:41.858697 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.858674    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c80976-1d79-48e5-8718-2695ae37d011-kube-api-access-mjnnk\" (UniqueName: \"kubernetes.io/projected/62c80976-1d79-48e5-8718-2695ae37d011-kube-api-access-mjnnk\") pod \"62c80976-1d79-48e5-8718-2695ae37d011\" (UID: \"62c80976-1d79-48e5-8718-2695ae37d011\") "
Oct 30 13:21:41.862156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.862128    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c80976-1d79-48e5-8718-2695ae37d011-openshift-service-ca" pod "62c80976-1d79-48e5-8718-2695ae37d011" (UID: "62c80976-1d79-48e5-8718-2695ae37d011"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:41.862339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.862209    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c80976-1d79-48e5-8718-2695ae37d011-metrics-certs" pod "62c80976-1d79-48e5-8718-2695ae37d011" (UID: "62c80976-1d79-48e5-8718-2695ae37d011"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:41.863687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.863665    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62c80976-1d79-48e5-8718-2695ae37d011-maas-api-tls" pod "62c80976-1d79-48e5-8718-2695ae37d011" (UID: "62c80976-1d79-48e5-8718-2695ae37d011"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:41.864070 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.864052    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c80976-1d79-48e5-8718-2695ae37d011-kube-api-access-mjnnk" pod "62c80976-1d79-48e5-8718-2695ae37d011" (UID: "62c80976-1d79-48e5-8718-2695ae37d011"). InnerVolumeSpecName "kube-api-access-mjnnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:41.961897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.961865    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62c80976-1d79-48e5-8718-2695ae37d011-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:41.961897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.961894    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62c80976-1d79-48e5-8718-2695ae37d011-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:41.961897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.961904    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62c80976-1d79-48e5-8718-2695ae37d011-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:41.962222 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:41.961913    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjnnk\" (UniqueName: \"kubernetes.io/projected/62c80976-1d79-48e5-8718-2695ae37d011-kube-api-access-mjnnk\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:42.225405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.225374    5893 generic.go:356] "Generic (PLEG): container finished" podID="62c80976-1d79-48e5-8718-2695ae37d011" containerID="365d2a330d543c11bd861b6c03ff496a23a92c322e10e9711a085c2f398c438f" exitCode=0
Oct 30 13:21:42.225604 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.225435    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"
Oct 30 13:21:42.225604 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.225468    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6" event={"ID":"62c80976-1d79-48e5-8718-2695ae37d011","Type":"ContainerDied","Data":"365d2a330d543c11bd861b6c03ff496a23a92c322e10e9711a085c2f398c438f"}
Oct 30 13:21:42.225604 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.225524    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6" event={"ID":"62c80976-1d79-48e5-8718-2695ae37d011","Type":"ContainerDied","Data":"c3b74d5843f768d5848130192e8a35a36baf65cd8ef3862d54c1c5b1e4416d1f"}
Oct 30 13:21:42.225604 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.225545    5893 scope.go:122] "RemoveContainer" containerID="365d2a330d543c11bd861b6c03ff496a23a92c322e10e9711a085c2f398c438f"
Oct 30 13:21:42.297566 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.297502    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"]
Oct 30 13:21:42.303989 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.303957    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-649303-5c6b5695bf-vx6p6"]
Oct 30 13:21:42.306230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.305781    5893 scope.go:122] "RemoveContainer" containerID="365d2a330d543c11bd861b6c03ff496a23a92c322e10e9711a085c2f398c438f"
Oct 30 13:21:42.306230 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:21:42.306220    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"365d2a330d543c11bd861b6c03ff496a23a92c322e10e9711a085c2f398c438f\": container with ID starting with 365d2a330d543c11bd861b6c03ff496a23a92c322e10e9711a085c2f398c438f not found: ID does not exist" containerID="365d2a330d543c11bd861b6c03ff496a23a92c322e10e9711a085c2f398c438f"
Oct 30 13:21:42.306474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.306254    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"365d2a330d543c11bd861b6c03ff496a23a92c322e10e9711a085c2f398c438f"} err="failed to get container status \"365d2a330d543c11bd861b6c03ff496a23a92c322e10e9711a085c2f398c438f\": rpc error: code = NotFound desc = could not find container \"365d2a330d543c11bd861b6c03ff496a23a92c322e10e9711a085c2f398c438f\": container with ID starting with 365d2a330d543c11bd861b6c03ff496a23a92c322e10e9711a085c2f398c438f not found: ID does not exist"
Oct 30 13:21:42.488387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.488355    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"
Oct 30 13:21:42.535312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.535266    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c80976-1d79-48e5-8718-2695ae37d011" path="/var/lib/kubelet/pods/62c80976-1d79-48e5-8718-2695ae37d011/volumes"
Oct 30 13:21:42.576795 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.576769    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc27afd0-973b-4e86-9ce1-adc6d294d7be-kube-api-access-ww47z\" (UniqueName: \"kubernetes.io/projected/dc27afd0-973b-4e86-9ce1-adc6d294d7be-kube-api-access-ww47z\") pod \"dc27afd0-973b-4e86-9ce1-adc6d294d7be\" (UID: \"dc27afd0-973b-4e86-9ce1-adc6d294d7be\") "
Oct 30 13:21:42.576989 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.576895    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc27afd0-973b-4e86-9ce1-adc6d294d7be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc27afd0-973b-4e86-9ce1-adc6d294d7be-plugins-config-volume\") pod \"dc27afd0-973b-4e86-9ce1-adc6d294d7be\" (UID: \"dc27afd0-973b-4e86-9ce1-adc6d294d7be\") "
Oct 30 13:21:42.577961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.577356    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc27afd0-973b-4e86-9ce1-adc6d294d7be-plugins-config-volume" pod "dc27afd0-973b-4e86-9ce1-adc6d294d7be" (UID: "dc27afd0-973b-4e86-9ce1-adc6d294d7be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:42.577961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.577942    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc27afd0-973b-4e86-9ce1-adc6d294d7be-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:42.579976 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.579952    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc27afd0-973b-4e86-9ce1-adc6d294d7be-kube-api-access-ww47z" pod "dc27afd0-973b-4e86-9ce1-adc6d294d7be" (UID: "dc27afd0-973b-4e86-9ce1-adc6d294d7be"). InnerVolumeSpecName "kube-api-access-ww47z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:42.679307 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:42.679253    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww47z\" (UniqueName: \"kubernetes.io/projected/dc27afd0-973b-4e86-9ce1-adc6d294d7be-kube-api-access-ww47z\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:43.236022 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:43.235989    5893 generic.go:356] "Generic (PLEG): container finished" podID="dc27afd0-973b-4e86-9ce1-adc6d294d7be" containerID="6a8a9507b0b5a4982d342fa59db603305d89065bd37279d6637b1893c5c2d1cd" exitCode=0
Oct 30 13:21:43.236238 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:43.236046    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"
Oct 30 13:21:43.236238 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:43.236054    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz" event={"ID":"dc27afd0-973b-4e86-9ce1-adc6d294d7be","Type":"ContainerDied","Data":"6a8a9507b0b5a4982d342fa59db603305d89065bd37279d6637b1893c5c2d1cd"}
Oct 30 13:21:43.236238 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:43.236097    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz" event={"ID":"dc27afd0-973b-4e86-9ce1-adc6d294d7be","Type":"ContainerDied","Data":"cce0e660d5c2c64dcc537c4fdf1606237ae021a896c9b60077fd48968e941f07"}
Oct 30 13:21:43.236238 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:43.236117    5893 scope.go:122] "RemoveContainer" containerID="6a8a9507b0b5a4982d342fa59db603305d89065bd37279d6637b1893c5c2d1cd"
Oct 30 13:21:43.261144 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:43.261124    5893 scope.go:122] "RemoveContainer" containerID="6a8a9507b0b5a4982d342fa59db603305d89065bd37279d6637b1893c5c2d1cd"
Oct 30 13:21:43.261607 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:21:43.261553    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a8a9507b0b5a4982d342fa59db603305d89065bd37279d6637b1893c5c2d1cd\": container with ID starting with 6a8a9507b0b5a4982d342fa59db603305d89065bd37279d6637b1893c5c2d1cd not found: ID does not exist" containerID="6a8a9507b0b5a4982d342fa59db603305d89065bd37279d6637b1893c5c2d1cd"
Oct 30 13:21:43.261721 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:43.261634    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a8a9507b0b5a4982d342fa59db603305d89065bd37279d6637b1893c5c2d1cd"} err="failed to get container status \"6a8a9507b0b5a4982d342fa59db603305d89065bd37279d6637b1893c5c2d1cd\": rpc error: code = NotFound desc = could not find container \"6a8a9507b0b5a4982d342fa59db603305d89065bd37279d6637b1893c5c2d1cd\": container with ID starting with 6a8a9507b0b5a4982d342fa59db603305d89065bd37279d6637b1893c5c2d1cd not found: ID does not exist"
Oct 30 13:21:43.298818 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:43.298787    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"]
Oct 30 13:21:43.310244 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:43.310212    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-6f9071-d78bbfb84-c6qpz"]
Oct 30 13:21:44.239494 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.239464    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"]
Oct 30 13:21:44.240465 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.240442    5893 cpu_manager.go:414] "containerMap: removing container" podUID="62c80976-1d79-48e5-8718-2695ae37d011" containerName="maas-api"
Oct 30 13:21:44.240465 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.240463    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c80976-1d79-48e5-8718-2695ae37d011" containerName="maas-api"
Oct 30 13:21:44.240679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.240499    5893 cpu_manager.go:414] "containerMap: removing container" podUID="dc27afd0-973b-4e86-9ce1-adc6d294d7be" containerName="payload-processing"
Oct 30 13:21:44.240679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.240507    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc27afd0-973b-4e86-9ce1-adc6d294d7be" containerName="payload-processing"
Oct 30 13:21:44.240782 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.240723    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c80976-1d79-48e5-8718-2695ae37d011" containerName="maas-api"
Oct 30 13:21:44.240782 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.240743    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc27afd0-973b-4e86-9ce1-adc6d294d7be" containerName="payload-processing"
Oct 30 13:21:44.244372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.244352    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:44.260320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.260293    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a5e47c-serving-cert\""
Oct 30 13:21:44.281030 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.281002    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"]
Oct 30 13:21:44.302631 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.302600    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:44.303067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.302655    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb52j\" (UniqueName: \"kubernetes.io/projected/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-kube-api-access-lb52j\") pod \"maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:44.303067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.302748    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-metrics-certs\") pod \"maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:44.303067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.302825    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-maas-api-tls\") pod \"maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:44.404688 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.404651    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-metrics-certs\") pod \"maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:44.404996 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.404723    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-maas-api-tls\") pod \"maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:44.404996 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.404897    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:44.404996 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.404926    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb52j\" (UniqueName: \"kubernetes.io/projected/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-kube-api-access-lb52j\") pod \"maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:44.405606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.405581    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:44.407516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.407495    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-maas-api-tls\") pod \"maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:44.407614 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.407529    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-metrics-certs\") pod \"maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:44.431828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.431806    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb52j\" (UniqueName: \"kubernetes.io/projected/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-kube-api-access-lb52j\") pod \"maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:44.536411 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.536323    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc27afd0-973b-4e86-9ce1-adc6d294d7be" path="/var/lib/kubelet/pods/dc27afd0-973b-4e86-9ce1-adc6d294d7be/volumes"
Oct 30 13:21:44.558858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.558831    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:44.877009 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:44.876982    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"]
Oct 30 13:21:44.879725 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:21:44.879698    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a4c9c9c_a0dd_4f38_979f_bc5567660a5f.slice/crio-1817d48b882d4cada67720da4ca0ef324b5ed255cde8d5e7f8a54d545e266784 WatchSource:0}: Error finding container 1817d48b882d4cada67720da4ca0ef324b5ed255cde8d5e7f8a54d545e266784: Status 404 returned error can't find the container with id 1817d48b882d4cada67720da4ca0ef324b5ed255cde8d5e7f8a54d545e266784
Oct 30 13:21:45.256962 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:45.256878    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89" event={"ID":"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f","Type":"ContainerStarted","Data":"1817d48b882d4cada67720da4ca0ef324b5ed255cde8d5e7f8a54d545e266784"}
Oct 30 13:21:46.283339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:46.283302    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89" event={"ID":"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f","Type":"ContainerStarted","Data":"f0d5f1ca355ad55936305d184d2f810ef9ebbf60bdb1a99bf2e84c043540dcc4"}
Oct 30 13:21:46.283830 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:46.283531    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:46.331677 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:46.331626    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89" podStartSLOduration=1.9980247100000001 podStartE2EDuration="2.331612059s" totalImagesPullingTime="333.587349ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:46.329820725 +0000 UTC m=+2147.913250420" watchObservedRunningTime="2026-10-30 13:21:46.331612059 +0000 UTC m=+2147.915041751"
Oct 30 13:21:48.647567 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:21:48.647527    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:21:49.788621 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:49.788587    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"]
Oct 30 13:21:49.789137 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:49.788872    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h" podUID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerName="istio-proxy" containerID="cri-o://db327b06fe8e5b5dd39104f58b5ad3f28153e1e6040b64b07b3f403dbe2db1e1" gracePeriod=30
Oct 30 13:21:52.299678 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:52.299643    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:21:53.214624 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.214591    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"]
Oct 30 13:21:53.218934 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.218912    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.245529 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.245488    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-acd125\"/\"default-dockercfg-hfjxl\""
Oct 30 13:21:53.246208 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.246176    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-acd125\"/\"kube-root-ca.crt\""
Oct 30 13:21:53.246395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.246326    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-acd125\"/\"openshift-service-ca.crt\""
Oct 30 13:21:53.251910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.251881    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"]
Oct 30 13:21:53.252038 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.252012    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-acd125\"/\"auth-test-model-acd125-kserve-self-signed-certs\""
Oct 30 13:21:53.390060 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.390029    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-model-cache\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.390601 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.390216    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-tmp-dir\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.390601 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.390430    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz2fq\" (UniqueName: \"kubernetes.io/projected/58298a7c-0dd3-474b-a5ae-263fae7c9e51-kube-api-access-wz2fq\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.390601 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.390491    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58298a7c-0dd3-474b-a5ae-263fae7c9e51-tls-certs\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.390601 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.390527    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-home\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.390601 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.390559    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-dshm\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.492125 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.492028    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz2fq\" (UniqueName: \"kubernetes.io/projected/58298a7c-0dd3-474b-a5ae-263fae7c9e51-kube-api-access-wz2fq\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.492125 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.492087    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58298a7c-0dd3-474b-a5ae-263fae7c9e51-tls-certs\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.492125 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.492121    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-home\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.492441 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.492144    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-dshm\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.492441 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.492231    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-model-cache\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.492441 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.492392    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-tmp-dir\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.492601 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.492556    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-home\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.492690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.492670    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-model-cache\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.492764 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.492748    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-tmp-dir\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.494753 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.494727    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-dshm\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.495079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.495062    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58298a7c-0dd3-474b-a5ae-263fae7c9e51-tls-certs\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.516245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.516209    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz2fq\" (UniqueName: \"kubernetes.io/projected/58298a7c-0dd3-474b-a5ae-263fae7c9e51-kube-api-access-wz2fq\") pod \"auth-test-model-acd125-kserve-5c5969b4fc-p6glw\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") " pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.535154 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.535125    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:53.909390 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:53.909364    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"]
Oct 30 13:21:53.910892 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:21:53.910857    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58298a7c_0dd3_474b_a5ae_263fae7c9e51.slice/crio-406a5a312d93463862e5271298677ad94d678cf5fd75a0fcba355b0bf997be69 WatchSource:0}: Error finding container 406a5a312d93463862e5271298677ad94d678cf5fd75a0fcba355b0bf997be69: Status 404 returned error can't find the container with id 406a5a312d93463862e5271298677ad94d678cf5fd75a0fcba355b0bf997be69
Oct 30 13:21:54.397214 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:54.397174    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw" event={"ID":"58298a7c-0dd3-474b-a5ae-263fae7c9e51","Type":"ContainerStarted","Data":"780db5298856c44ae984b3252efc810efeb97cac71e8576eb6efcbebd2f8012b"}
Oct 30 13:21:54.397709 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:54.397254    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw" event={"ID":"58298a7c-0dd3-474b-a5ae-263fae7c9e51","Type":"ContainerStarted","Data":"406a5a312d93463862e5271298677ad94d678cf5fd75a0fcba355b0bf997be69"}
Oct 30 13:21:54.442801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:54.442745    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw" podStartSLOduration=1.442725232 podStartE2EDuration="1.442725232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:54.438663772 +0000 UTC m=+2156.022093484" watchObservedRunningTime="2026-10-30 13:21:54.442725232 +0000 UTC m=+2156.026154935"
Oct 30 13:21:54.535412 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:54.535386    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:21:55.092331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.092307    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:21:55.126555 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.126523    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-credential-socket\") pod \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") "
Oct 30 13:21:55.126756 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.126697    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-envoy\") pod \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") "
Oct 30 13:21:55.126756 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.126732    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-workload-socket\") pod \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") "
Oct 30 13:21:55.126892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.126779    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istiod-ca-cert\") pod \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") "
Oct 30 13:21:55.126892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.126834    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-kube-api-access-hhc9s\" (UniqueName: \"kubernetes.io/projected/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-kube-api-access-hhc9s\") pod \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") "
Oct 30 13:21:55.127000 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.126921    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-workload-certs\") pod \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") "
Oct 30 13:21:55.127000 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.126964    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-podinfo\") pod \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") "
Oct 30 13:21:55.127000 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.126972    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-credential-socket" pod "1ebcad4e-6749-4f92-9f85-6e72eb1a1954" (UID: "1ebcad4e-6749-4f92-9f85-6e72eb1a1954"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:55.127164 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.127035    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-data\") pod \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") "
Oct 30 13:21:55.127164 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.127070    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-token\" (UniqueName: \"kubernetes.io/projected/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-token\") pod \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\" (UID: \"1ebcad4e-6749-4f92-9f85-6e72eb1a1954\") "
Oct 30 13:21:55.127317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.127138    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-workload-socket" pod "1ebcad4e-6749-4f92-9f85-6e72eb1a1954" (UID: "1ebcad4e-6749-4f92-9f85-6e72eb1a1954"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:55.127317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.127243    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-workload-certs" pod "1ebcad4e-6749-4f92-9f85-6e72eb1a1954" (UID: "1ebcad4e-6749-4f92-9f85-6e72eb1a1954"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:55.127449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.127392    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-data" pod "1ebcad4e-6749-4f92-9f85-6e72eb1a1954" (UID: "1ebcad4e-6749-4f92-9f85-6e72eb1a1954"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:55.127839 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.127815    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:55.128012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.127997    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:55.128156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.128141    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:55.128287 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.128258    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:55.128551 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.128006    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istiod-ca-cert" pod "1ebcad4e-6749-4f92-9f85-6e72eb1a1954" (UID: "1ebcad4e-6749-4f92-9f85-6e72eb1a1954"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:55.130007 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.129954    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-podinfo" pod "1ebcad4e-6749-4f92-9f85-6e72eb1a1954" (UID: "1ebcad4e-6749-4f92-9f85-6e72eb1a1954"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:21:55.130128 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.130001    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-envoy" pod "1ebcad4e-6749-4f92-9f85-6e72eb1a1954" (UID: "1ebcad4e-6749-4f92-9f85-6e72eb1a1954"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:21:55.130306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.130250    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-token" pod "1ebcad4e-6749-4f92-9f85-6e72eb1a1954" (UID: "1ebcad4e-6749-4f92-9f85-6e72eb1a1954"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:55.130395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.130368    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-kube-api-access-hhc9s" pod "1ebcad4e-6749-4f92-9f85-6e72eb1a1954" (UID: "1ebcad4e-6749-4f92-9f85-6e72eb1a1954"). InnerVolumeSpecName "kube-api-access-hhc9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:55.230255 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.230222    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhc9s\" (UniqueName: \"kubernetes.io/projected/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-kube-api-access-hhc9s\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:55.230255 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.230255    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:55.230488 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.230267    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:55.230488 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.230318    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:55.230488 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.230330    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ebcad4e-6749-4f92-9f85-6e72eb1a1954-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:21:55.408068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.407979    5893 generic.go:356] "Generic (PLEG): container finished" podID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerID="db327b06fe8e5b5dd39104f58b5ad3f28153e1e6040b64b07b3f403dbe2db1e1" exitCode=0
Oct 30 13:21:55.408540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.408082    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"
Oct 30 13:21:55.408540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.408107    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h" event={"ID":"1ebcad4e-6749-4f92-9f85-6e72eb1a1954","Type":"ContainerDied","Data":"db327b06fe8e5b5dd39104f58b5ad3f28153e1e6040b64b07b3f403dbe2db1e1"}
Oct 30 13:21:55.408540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.408172    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h" event={"ID":"1ebcad4e-6749-4f92-9f85-6e72eb1a1954","Type":"ContainerDied","Data":"0d387daf50bc35979bf33ad646c6480ee105ee20a6bf60136a54d652ba145760"}
Oct 30 13:21:55.408540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.408196    5893 scope.go:122] "RemoveContainer" containerID="db327b06fe8e5b5dd39104f58b5ad3f28153e1e6040b64b07b3f403dbe2db1e1"
Oct 30 13:21:55.420218 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.420198    5893 scope.go:122] "RemoveContainer" containerID="df7f17ae5f441d1ee4b66daffbcd1670bd2ce1084125d4ed86ef10bf1ab5d9e3"
Oct 30 13:21:55.431357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.431336    5893 scope.go:122] "RemoveContainer" containerID="db327b06fe8e5b5dd39104f58b5ad3f28153e1e6040b64b07b3f403dbe2db1e1"
Oct 30 13:21:55.431634 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:21:55.431613    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db327b06fe8e5b5dd39104f58b5ad3f28153e1e6040b64b07b3f403dbe2db1e1\": container with ID starting with db327b06fe8e5b5dd39104f58b5ad3f28153e1e6040b64b07b3f403dbe2db1e1 not found: ID does not exist" containerID="db327b06fe8e5b5dd39104f58b5ad3f28153e1e6040b64b07b3f403dbe2db1e1"
Oct 30 13:21:55.431691 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.431646    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db327b06fe8e5b5dd39104f58b5ad3f28153e1e6040b64b07b3f403dbe2db1e1"} err="failed to get container status \"db327b06fe8e5b5dd39104f58b5ad3f28153e1e6040b64b07b3f403dbe2db1e1\": rpc error: code = NotFound desc = could not find container \"db327b06fe8e5b5dd39104f58b5ad3f28153e1e6040b64b07b3f403dbe2db1e1\": container with ID starting with db327b06fe8e5b5dd39104f58b5ad3f28153e1e6040b64b07b3f403dbe2db1e1 not found: ID does not exist"
Oct 30 13:21:55.431691 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.431669    5893 scope.go:122] "RemoveContainer" containerID="df7f17ae5f441d1ee4b66daffbcd1670bd2ce1084125d4ed86ef10bf1ab5d9e3"
Oct 30 13:21:55.431943 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:21:55.431924    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7f17ae5f441d1ee4b66daffbcd1670bd2ce1084125d4ed86ef10bf1ab5d9e3\": container with ID starting with df7f17ae5f441d1ee4b66daffbcd1670bd2ce1084125d4ed86ef10bf1ab5d9e3 not found: ID does not exist" containerID="df7f17ae5f441d1ee4b66daffbcd1670bd2ce1084125d4ed86ef10bf1ab5d9e3"
Oct 30 13:21:55.431985 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.431949    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7f17ae5f441d1ee4b66daffbcd1670bd2ce1084125d4ed86ef10bf1ab5d9e3"} err="failed to get container status \"df7f17ae5f441d1ee4b66daffbcd1670bd2ce1084125d4ed86ef10bf1ab5d9e3\": rpc error: code = NotFound desc = could not find container \"df7f17ae5f441d1ee4b66daffbcd1670bd2ce1084125d4ed86ef10bf1ab5d9e3\": container with ID starting with df7f17ae5f441d1ee4b66daffbcd1670bd2ce1084125d4ed86ef10bf1ab5d9e3 not found: ID does not exist"
Oct 30 13:21:55.465498 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.465451    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"]
Oct 30 13:21:55.477303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:55.477259    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-649303-openshift-default-85fbcccd4c-rk22h"]
Oct 30 13:21:56.551027 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:56.550992    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" path="/var/lib/kubelet/pods/1ebcad4e-6749-4f92-9f85-6e72eb1a1954/volumes"
Oct 30 13:21:59.839900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:21:59.839870    5893 scope.go:122] "RemoveContainer" containerID="c57a16a9dba0f43d11f1ba6c0f684cdd48e045e44ddd391a40be6bb24653d2fd"
Oct 30 13:22:03.535337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:03.535299    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:22:03.551194 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:03.551164    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:22:04.496957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:04.496925    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:22:06.495967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:06.495933    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"]
Oct 30 13:22:06.497946 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:06.497910    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr" podUID="079c739c-702b-4eeb-9e5e-b30ad98b8fd2" containerName="istio-proxy" containerID="cri-o://117b0791438fc3c50cac6b93a3a18acbc2475f1d1cad45d984e16e7005eac806" gracePeriod=30
Oct 30 13:22:11.810953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.810927    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:22:11.923673 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.923631    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istiod-ca-cert\") pod \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") "
Oct 30 13:22:11.923902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.923697    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-workload-certs\") pod \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") "
Oct 30 13:22:11.923902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.923758    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-envoy\") pod \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") "
Oct 30 13:22:11.923902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.923799    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-kube-api-access-4x6nw\" (UniqueName: \"kubernetes.io/projected/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-kube-api-access-4x6nw\") pod \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") "
Oct 30 13:22:11.923902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.923846    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-data\") pod \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") "
Oct 30 13:22:11.923902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.923891    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-podinfo\") pod \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") "
Oct 30 13:22:11.924201 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.923963    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-token\" (UniqueName: \"kubernetes.io/projected/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-token\") pod \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") "
Oct 30 13:22:11.924259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.924223    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-workload-socket\") pod \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") "
Oct 30 13:22:11.924383 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.924364    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-credential-socket\") pod \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\" (UID: \"079c739c-702b-4eeb-9e5e-b30ad98b8fd2\") "
Oct 30 13:22:11.925395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.925367    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-credential-socket" pod "079c739c-702b-4eeb-9e5e-b30ad98b8fd2" (UID: "079c739c-702b-4eeb-9e5e-b30ad98b8fd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:11.925748 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.925725    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istiod-ca-cert" pod "079c739c-702b-4eeb-9e5e-b30ad98b8fd2" (UID: "079c739c-702b-4eeb-9e5e-b30ad98b8fd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:22:11.925998 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.925977    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-workload-certs" pod "079c739c-702b-4eeb-9e5e-b30ad98b8fd2" (UID: "079c739c-702b-4eeb-9e5e-b30ad98b8fd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:11.926630 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.926598    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-workload-socket" pod "079c739c-702b-4eeb-9e5e-b30ad98b8fd2" (UID: "079c739c-702b-4eeb-9e5e-b30ad98b8fd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:11.926630 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.926609    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-data" pod "079c739c-702b-4eeb-9e5e-b30ad98b8fd2" (UID: "079c739c-702b-4eeb-9e5e-b30ad98b8fd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:11.929141 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.929111    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-token" pod "079c739c-702b-4eeb-9e5e-b30ad98b8fd2" (UID: "079c739c-702b-4eeb-9e5e-b30ad98b8fd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:11.929141 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.929127    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-envoy" pod "079c739c-702b-4eeb-9e5e-b30ad98b8fd2" (UID: "079c739c-702b-4eeb-9e5e-b30ad98b8fd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:22:11.929266 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.929213    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-kube-api-access-4x6nw" pod "079c739c-702b-4eeb-9e5e-b30ad98b8fd2" (UID: "079c739c-702b-4eeb-9e5e-b30ad98b8fd2"). InnerVolumeSpecName "kube-api-access-4x6nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:11.930941 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:11.930913    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-podinfo" pod "079c739c-702b-4eeb-9e5e-b30ad98b8fd2" (UID: "079c739c-702b-4eeb-9e5e-b30ad98b8fd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:22:12.026230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.026197    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:22:12.026230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.026224    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:22:12.026230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.026233    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:22:12.026490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.026243    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:22:12.026490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.026252    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:22:12.026490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.026260    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x6nw\" (UniqueName: \"kubernetes.io/projected/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-kube-api-access-4x6nw\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:22:12.026490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.026301    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:22:12.026490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.026310    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:22:12.026490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.026316    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079c739c-702b-4eeb-9e5e-b30ad98b8fd2-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:22:12.557513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.557477    5893 generic.go:356] "Generic (PLEG): container finished" podID="079c739c-702b-4eeb-9e5e-b30ad98b8fd2" containerID="117b0791438fc3c50cac6b93a3a18acbc2475f1d1cad45d984e16e7005eac806" exitCode=0
Oct 30 13:22:12.557513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.557541    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"
Oct 30 13:22:12.557513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.557569    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr" event={"ID":"079c739c-702b-4eeb-9e5e-b30ad98b8fd2","Type":"ContainerDied","Data":"117b0791438fc3c50cac6b93a3a18acbc2475f1d1cad45d984e16e7005eac806"}
Oct 30 13:22:12.557513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.557654    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr" event={"ID":"079c739c-702b-4eeb-9e5e-b30ad98b8fd2","Type":"ContainerDied","Data":"89b4e30a46d015190c7049bda029aa6b66db2e756f362a054cdc0575630b4c49"}
Oct 30 13:22:12.557513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.557703    5893 scope.go:122] "RemoveContainer" containerID="117b0791438fc3c50cac6b93a3a18acbc2475f1d1cad45d984e16e7005eac806"
Oct 30 13:22:12.576730 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.576699    5893 scope.go:122] "RemoveContainer" containerID="117b0791438fc3c50cac6b93a3a18acbc2475f1d1cad45d984e16e7005eac806"
Oct 30 13:22:12.577440 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:22:12.577391    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"117b0791438fc3c50cac6b93a3a18acbc2475f1d1cad45d984e16e7005eac806\": container with ID starting with 117b0791438fc3c50cac6b93a3a18acbc2475f1d1cad45d984e16e7005eac806 not found: ID does not exist" containerID="117b0791438fc3c50cac6b93a3a18acbc2475f1d1cad45d984e16e7005eac806"
Oct 30 13:22:12.577551 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.577456    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"117b0791438fc3c50cac6b93a3a18acbc2475f1d1cad45d984e16e7005eac806"} err="failed to get container status \"117b0791438fc3c50cac6b93a3a18acbc2475f1d1cad45d984e16e7005eac806\": rpc error: code = NotFound desc = could not find container \"117b0791438fc3c50cac6b93a3a18acbc2475f1d1cad45d984e16e7005eac806\": container with ID starting with 117b0791438fc3c50cac6b93a3a18acbc2475f1d1cad45d984e16e7005eac806 not found: ID does not exist"
Oct 30 13:22:12.633138 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.633099    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"]
Oct 30 13:22:12.644420 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:12.644389    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6f9071-openshift-default-657cb56d95-rcjvr"]
Oct 30 13:22:14.536396 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:14.536363    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="079c739c-702b-4eeb-9e5e-b30ad98b8fd2" path="/var/lib/kubelet/pods/079c739c-702b-4eeb-9e5e-b30ad98b8fd2/volumes"
Oct 30 13:22:18.647333 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:22:18.647293    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:22:43.902565 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:43.902532    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"]
Oct 30 13:22:43.932787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:43.932755    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"]
Oct 30 13:22:46.819507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.819477    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"]
Oct 30 13:22:46.820372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.820350    5893 cpu_manager.go:414] "containerMap: removing container" podUID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerName="istio-proxy"
Oct 30 13:22:46.820372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.820370    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerName="istio-proxy"
Oct 30 13:22:46.820558 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.820402    5893 cpu_manager.go:414] "containerMap: removing container" podUID="079c739c-702b-4eeb-9e5e-b30ad98b8fd2" containerName="istio-proxy"
Oct 30 13:22:46.820558 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.820408    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079c739c-702b-4eeb-9e5e-b30ad98b8fd2" containerName="istio-proxy"
Oct 30 13:22:46.820558 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.820539    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="079c739c-702b-4eeb-9e5e-b30ad98b8fd2" containerName="istio-proxy"
Oct 30 13:22:46.820558 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.820554    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerName="istio-proxy"
Oct 30 13:22:46.825535 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.825511    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:46.853160 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.853133    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-e9d641\"/\"default-dockercfg-rwklk\""
Oct 30 13:22:46.853525 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.853465    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-e9d641\"/\"openshift-service-ca.crt\""
Oct 30 13:22:46.854043 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.854020    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-e9d641\"/\"kube-root-ca.crt\""
Oct 30 13:22:46.854209 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.854185    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-e9d641\"/\"auth-test-model-e9d641-kserve-self-signed-certs\""
Oct 30 13:22:46.858585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.858559    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"]
Oct 30 13:22:46.946123 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.946084    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-dshm\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:46.946362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.946227    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gm2j\" (UniqueName: \"kubernetes.io/projected/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-kube-api-access-6gm2j\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:46.946362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.946303    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-tmp-dir\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:46.946493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.946417    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-model-cache\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:46.946671 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.946624    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-tls-certs\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:46.946961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:46.946942    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-home\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:47.048459 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.048420    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-model-cache\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:47.048665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.048490    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-tls-certs\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:47.048665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.048579    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-home\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:47.048787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.048704    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-dshm\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:47.048787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.048742    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gm2j\" (UniqueName: \"kubernetes.io/projected/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-kube-api-access-6gm2j\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:47.048787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.048777    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-tmp-dir\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:47.048946 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.048886    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-model-cache\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:47.049091 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.049071    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-home\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:47.049255 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.049235    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-tmp-dir\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:47.051287 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.051251    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-tls-certs\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:47.051460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.051442    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-dshm\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:47.074562 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.074484    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gm2j\" (UniqueName: \"kubernetes.io/projected/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-kube-api-access-6gm2j\") pod \"auth-test-model-e9d641-kserve-855bb7d4f-p9sxg\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") " pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:47.149293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.147581    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:47.533333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.530983    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"]
Oct 30 13:22:47.609335 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:22:47.608783    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52ba1504_83f4_4592_9b3e_e5be4ba8e20a.slice/crio-9376bc9b994dffe7e7927c7be30186e433e003650f8e1177f42384e2e63faa99 WatchSource:0}: Error finding container 9376bc9b994dffe7e7927c7be30186e433e003650f8e1177f42384e2e63faa99: Status 404 returned error can't find the container with id 9376bc9b994dffe7e7927c7be30186e433e003650f8e1177f42384e2e63faa99
Oct 30 13:22:47.871405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.871307    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg" event={"ID":"52ba1504-83f4-4592-9b3e-e5be4ba8e20a","Type":"ContainerStarted","Data":"6077defc75db97b0d3cf0a60e42aa665d8c8f3719520e04f79fce6669eb4fa22"}
Oct 30 13:22:47.871405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.871354    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg" event={"ID":"52ba1504-83f4-4592-9b3e-e5be4ba8e20a","Type":"ContainerStarted","Data":"9376bc9b994dffe7e7927c7be30186e433e003650f8e1177f42384e2e63faa99"}
Oct 30 13:22:47.913072 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:47.913016    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg" podStartSLOduration=1.9130014260000001 podStartE2EDuration="1.913001426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:47.912166495 +0000 UTC m=+2209.495596188" watchObservedRunningTime="2026-10-30 13:22:47.913001426 +0000 UTC m=+2209.496431118"
Oct 30 13:22:48.148385 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:48.148286    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:48.655932 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:22:48.655897    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:22:48.903258 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:48.903220    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m" podUID="ce126ff4-a811-4d1e-83d0-3037add33d2e" containerName="payload-processing" containerID="cri-o://55c11d7d7dd67d6cb2f6f819a2baa723704873d53143576eb467cd79aa309cfd" gracePeriod=25
Oct 30 13:22:48.933151 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:48.933048    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r" podUID="42c820c0-bd02-40ac-b278-8974ade564b4" containerName="payload-pre-processing" containerID="cri-o://87689cb355dd3766836d309aa39144f503e744af4fc056705f1ada6758d9a78a" gracePeriod=25
Oct 30 13:22:50.472602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:50.472557    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m" podUID="ce126ff4-a811-4d1e-83d0-3037add33d2e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:22:51.492233 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:51.492187    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r" podUID="42c820c0-bd02-40ac-b278-8974ade564b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:22:57.148174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:57.148135    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:57.160858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:57.160833    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:22:57.964262 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:22:57.964233    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:23:00.472192 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:00.472148    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m" podUID="ce126ff4-a811-4d1e-83d0-3037add33d2e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:23:01.492122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:01.492089    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r" podUID="42c820c0-bd02-40ac-b278-8974ade564b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:23:09.116116 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.115761    5893 generic.go:356] "Generic (PLEG): container finished" podID="42c820c0-bd02-40ac-b278-8974ade564b4" containerID="87689cb355dd3766836d309aa39144f503e744af4fc056705f1ada6758d9a78a" exitCode=0
Oct 30 13:23:09.116116 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.116025    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r" event={"ID":"42c820c0-bd02-40ac-b278-8974ade564b4","Type":"ContainerDied","Data":"87689cb355dd3766836d309aa39144f503e744af4fc056705f1ada6758d9a78a"}
Oct 30 13:23:09.118898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.118778    5893 generic.go:356] "Generic (PLEG): container finished" podID="ce126ff4-a811-4d1e-83d0-3037add33d2e" containerID="55c11d7d7dd67d6cb2f6f819a2baa723704873d53143576eb467cd79aa309cfd" exitCode=0
Oct 30 13:23:09.118898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.118827    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m" event={"ID":"ce126ff4-a811-4d1e-83d0-3037add33d2e","Type":"ContainerDied","Data":"55c11d7d7dd67d6cb2f6f819a2baa723704873d53143576eb467cd79aa309cfd"}
Oct 30 13:23:09.256457 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.256433    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"
Oct 30 13:23:09.286590 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.286562    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"
Oct 30 13:23:09.361006 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.360963    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce126ff4-a811-4d1e-83d0-3037add33d2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce126ff4-a811-4d1e-83d0-3037add33d2e-plugins-config-volume\") pod \"ce126ff4-a811-4d1e-83d0-3037add33d2e\" (UID: \"ce126ff4-a811-4d1e-83d0-3037add33d2e\") "
Oct 30 13:23:09.363594 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.363562    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce126ff4-a811-4d1e-83d0-3037add33d2e-kube-api-access-d2h78\" (UniqueName: \"kubernetes.io/projected/ce126ff4-a811-4d1e-83d0-3037add33d2e-kube-api-access-d2h78\") pod \"ce126ff4-a811-4d1e-83d0-3037add33d2e\" (UID: \"ce126ff4-a811-4d1e-83d0-3037add33d2e\") "
Oct 30 13:23:09.363742 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.363694    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42c820c0-bd02-40ac-b278-8974ade564b4-kube-api-access-rwxsh\" (UniqueName: \"kubernetes.io/projected/42c820c0-bd02-40ac-b278-8974ade564b4-kube-api-access-rwxsh\") pod \"42c820c0-bd02-40ac-b278-8974ade564b4\" (UID: \"42c820c0-bd02-40ac-b278-8974ade564b4\") "
Oct 30 13:23:09.363742 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.363733    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42c820c0-bd02-40ac-b278-8974ade564b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c820c0-bd02-40ac-b278-8974ade564b4-plugins-config-volume\") pod \"42c820c0-bd02-40ac-b278-8974ade564b4\" (UID: \"42c820c0-bd02-40ac-b278-8974ade564b4\") "
Oct 30 13:23:09.367429 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.362740    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce126ff4-a811-4d1e-83d0-3037add33d2e-plugins-config-volume" pod "ce126ff4-a811-4d1e-83d0-3037add33d2e" (UID: "ce126ff4-a811-4d1e-83d0-3037add33d2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:23:09.367607 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.367400    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c820c0-bd02-40ac-b278-8974ade564b4-plugins-config-volume" pod "42c820c0-bd02-40ac-b278-8974ade564b4" (UID: "42c820c0-bd02-40ac-b278-8974ade564b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:23:09.383509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.383469    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c820c0-bd02-40ac-b278-8974ade564b4-kube-api-access-rwxsh" pod "42c820c0-bd02-40ac-b278-8974ade564b4" (UID: "42c820c0-bd02-40ac-b278-8974ade564b4"). InnerVolumeSpecName "kube-api-access-rwxsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:09.383701 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.383504    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce126ff4-a811-4d1e-83d0-3037add33d2e-kube-api-access-d2h78" pod "ce126ff4-a811-4d1e-83d0-3037add33d2e" (UID: "ce126ff4-a811-4d1e-83d0-3037add33d2e"). InnerVolumeSpecName "kube-api-access-d2h78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:09.470383 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.467739    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwxsh\" (UniqueName: \"kubernetes.io/projected/42c820c0-bd02-40ac-b278-8974ade564b4-kube-api-access-rwxsh\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:23:09.470383 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.467792    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42c820c0-bd02-40ac-b278-8974ade564b4-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:23:09.470383 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.467809    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce126ff4-a811-4d1e-83d0-3037add33d2e-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:23:09.470383 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:09.467829    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2h78\" (UniqueName: \"kubernetes.io/projected/ce126ff4-a811-4d1e-83d0-3037add33d2e-kube-api-access-d2h78\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:23:10.127034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:10.127004    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"
Oct 30 13:23:10.127034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:10.127025    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m" event={"ID":"ce126ff4-a811-4d1e-83d0-3037add33d2e","Type":"ContainerDied","Data":"b137cd8c5210f68e2640c79656d4f3924cc2e611caa4280a09b0ba49ee320b65"}
Oct 30 13:23:10.127600 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:10.127094    5893 scope.go:122] "RemoveContainer" containerID="55c11d7d7dd67d6cb2f6f819a2baa723704873d53143576eb467cd79aa309cfd"
Oct 30 13:23:10.128964 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:10.128940    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r" event={"ID":"42c820c0-bd02-40ac-b278-8974ade564b4","Type":"ContainerDied","Data":"7ef6847632ab013242ccf3c4b19d5a86efe8e0f6000e0b58d36bfdfe64a2dcc7"}
Oct 30 13:23:10.129066 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:10.128946    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"
Oct 30 13:23:10.140149 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:10.140061    5893 scope.go:122] "RemoveContainer" containerID="87689cb355dd3766836d309aa39144f503e744af4fc056705f1ada6758d9a78a"
Oct 30 13:23:10.204929 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:10.204846    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"]
Oct 30 13:23:10.215941 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:10.215911    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6af611-66b65b6d95-trk8r"]
Oct 30 13:23:10.244502 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:10.244471    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"]
Oct 30 13:23:10.249191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:10.249163    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6af611-66b99cd54-stg4m"]
Oct 30 13:23:10.535155 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:10.535114    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42c820c0-bd02-40ac-b278-8974ade564b4" path="/var/lib/kubelet/pods/42c820c0-bd02-40ac-b278-8974ade564b4/volumes"
Oct 30 13:23:10.535752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:10.535729    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce126ff4-a811-4d1e-83d0-3037add33d2e" path="/var/lib/kubelet/pods/ce126ff4-a811-4d1e-83d0-3037add33d2e/volumes"
Oct 30 13:23:14.595934 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.595902    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"]
Oct 30 13:23:14.596737 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.596722    5893 cpu_manager.go:414] "containerMap: removing container" podUID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerName="istio-proxy"
Oct 30 13:23:14.596788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.596738    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerName="istio-proxy"
Oct 30 13:23:14.596788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.596756    5893 cpu_manager.go:414] "containerMap: removing container" podUID="ce126ff4-a811-4d1e-83d0-3037add33d2e" containerName="payload-processing"
Oct 30 13:23:14.596788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.596763    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce126ff4-a811-4d1e-83d0-3037add33d2e" containerName="payload-processing"
Oct 30 13:23:14.596898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.596793    5893 cpu_manager.go:414] "containerMap: removing container" podUID="42c820c0-bd02-40ac-b278-8974ade564b4" containerName="payload-pre-processing"
Oct 30 13:23:14.596898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.596799    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42c820c0-bd02-40ac-b278-8974ade564b4" containerName="payload-pre-processing"
Oct 30 13:23:14.596965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.596904    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ebcad4e-6749-4f92-9f85-6e72eb1a1954" containerName="istio-proxy"
Oct 30 13:23:14.596965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.596916    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce126ff4-a811-4d1e-83d0-3037add33d2e" containerName="payload-processing"
Oct 30 13:23:14.596965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.596928    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="42c820c0-bd02-40ac-b278-8974ade564b4" containerName="payload-pre-processing"
Oct 30 13:23:14.602480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.602458    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.616310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.614102    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"]
Oct 30 13:23:14.621093 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.621067    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2b8461\"/\"openshift-service-ca.crt\""
Oct 30 13:23:14.621093 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.621089    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2b8461\"/\"kube-root-ca.crt\""
Oct 30 13:23:14.631098 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.631061    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2b8461\"/\"default-dockercfg-sgtrf\""
Oct 30 13:23:14.631242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.631214    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2b8461\"/\"test-model-2b8461-kserve-self-signed-certs\""
Oct 30 13:23:14.664205 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.664173    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-home\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.664205 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.664208    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-model-cache\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.664477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.664259    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-tmp-dir\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.664477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.664373    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00359ed8-1f53-451e-8095-a5c4cedeb18b-tls-certs\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.664588 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.664505    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgmx5\" (UniqueName: \"kubernetes.io/projected/00359ed8-1f53-451e-8095-a5c4cedeb18b-kube-api-access-bgmx5\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.664588 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.664546    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-dshm\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.766369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.766326    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgmx5\" (UniqueName: \"kubernetes.io/projected/00359ed8-1f53-451e-8095-a5c4cedeb18b-kube-api-access-bgmx5\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.766560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.766394    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-dshm\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.766560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.766506    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-home\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.766560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.766531    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-model-cache\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.766729 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.766606    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-tmp-dir\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.766729 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.766668    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00359ed8-1f53-451e-8095-a5c4cedeb18b-tls-certs\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.767237 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.767182    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-model-cache\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.767237 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.767210    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-home\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.767942 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.767719    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-tmp-dir\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.769755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.769733    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-dshm\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.770154 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.770093    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00359ed8-1f53-451e-8095-a5c4cedeb18b-tls-certs\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.799625 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.799593    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgmx5\" (UniqueName: \"kubernetes.io/projected/00359ed8-1f53-451e-8095-a5c4cedeb18b-kube-api-access-bgmx5\") pod \"test-model-2b8461-kserve-574c5bfbb9-bj8gf\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") " pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:14.918690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:14.918603    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:15.266116 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:15.266084    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"]
Oct 30 13:23:15.267487 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:23:15.267458    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00359ed8_1f53_451e_8095_a5c4cedeb18b.slice/crio-4564ce3487db523bf8aae96da6889509a2bae7750b1a3ea61f1e2c2e96f9fb75 WatchSource:0}: Error finding container 4564ce3487db523bf8aae96da6889509a2bae7750b1a3ea61f1e2c2e96f9fb75: Status 404 returned error can't find the container with id 4564ce3487db523bf8aae96da6889509a2bae7750b1a3ea61f1e2c2e96f9fb75
Oct 30 13:23:16.194958 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:16.194926    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf" event={"ID":"00359ed8-1f53-451e-8095-a5c4cedeb18b","Type":"ContainerStarted","Data":"f247fabbdd6071416554b0187ebc9269707cae72415b21ae95aaaae2e972914f"}
Oct 30 13:23:16.194958 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:16.194969    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf" event={"ID":"00359ed8-1f53-451e-8095-a5c4cedeb18b","Type":"ContainerStarted","Data":"4564ce3487db523bf8aae96da6889509a2bae7750b1a3ea61f1e2c2e96f9fb75"}
Oct 30 13:23:16.270801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:16.270746    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf" podStartSLOduration=2.2707328589999998 podStartE2EDuration="2.270732859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:16.270502266 +0000 UTC m=+2237.853931961" watchObservedRunningTime="2026-10-30 13:23:16.270732859 +0000 UTC m=+2237.854162552"
Oct 30 13:23:16.921011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:16.920980    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:18.666514 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:23:18.666479    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:23:22.322835 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:22.322802    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"]
Oct 30 13:23:22.360659 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:22.360627    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"]
Oct 30 13:23:23.424181 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.424146    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"]
Oct 30 13:23:23.430913 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.430887    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.443418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.443389    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"]
Oct 30 13:23:23.450657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.450514    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a5e47c\"/\"kube-root-ca.crt\""
Oct 30 13:23:23.450657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.450583    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a5e47c\"/\"default-dockercfg-hls2g\""
Oct 30 13:23:23.460405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.460372    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-a5e47c\"/\"openshift-service-ca.crt\""
Oct 30 13:23:23.460405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.460404    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-a5e47c\"/\"test-model-a5e47c-kserve-self-signed-certs\""
Oct 30 13:23:23.522059 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.522028    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-dshm\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.522302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.522176    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-tmp-dir\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.522393 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.522359    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/166ae2e1-30dd-46e5-b25f-51f272d13bca-tls-certs\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.522592 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.522576    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-home\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.522650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.522602    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5shc2\" (UniqueName: \"kubernetes.io/projected/166ae2e1-30dd-46e5-b25f-51f272d13bca-kube-api-access-5shc2\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.522776 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.522762    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-model-cache\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.624848 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.624810    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-model-cache\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.625139 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.625110    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-dshm\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.625318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.625293    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-model-cache\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.625790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.625770    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-tmp-dir\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.626017 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.625992    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/166ae2e1-30dd-46e5-b25f-51f272d13bca-tls-certs\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.626119 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.626041    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-tmp-dir\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.626186 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.626166    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-home\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.626243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.626195    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5shc2\" (UniqueName: \"kubernetes.io/projected/166ae2e1-30dd-46e5-b25f-51f272d13bca-kube-api-access-5shc2\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.626540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.626516    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-home\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.628403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.628381    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-dshm\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.628582 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.628562    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/166ae2e1-30dd-46e5-b25f-51f272d13bca-tls-certs\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.651049 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.651018    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5shc2\" (UniqueName: \"kubernetes.io/projected/166ae2e1-30dd-46e5-b25f-51f272d13bca-kube-api-access-5shc2\") pod \"test-model-a5e47c-kserve-59ccf4d597-fjj68\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") " pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:23.751303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:23.750659    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:24.110552 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:24.110459    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"]
Oct 30 13:23:24.118843 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:23:24.118811    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod166ae2e1_30dd_46e5_b25f_51f272d13bca.slice/crio-fadae60a02ee30eb3e279cd94df976857a677ab16c895dcf72d59c0881e281c4 WatchSource:0}: Error finding container fadae60a02ee30eb3e279cd94df976857a677ab16c895dcf72d59c0881e281c4: Status 404 returned error can't find the container with id fadae60a02ee30eb3e279cd94df976857a677ab16c895dcf72d59c0881e281c4
Oct 30 13:23:24.327296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:24.326834    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68" event={"ID":"166ae2e1-30dd-46e5-b25f-51f272d13bca","Type":"ContainerStarted","Data":"2648ec65a4357eaf28a125d584831c2c9ed59fd4f2910667faa3ca5e093aa61b"}
Oct 30 13:23:24.327296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:24.326883    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68" event={"ID":"166ae2e1-30dd-46e5-b25f-51f272d13bca","Type":"ContainerStarted","Data":"fadae60a02ee30eb3e279cd94df976857a677ab16c895dcf72d59c0881e281c4"}
Oct 30 13:23:24.389766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:24.389648    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68" podStartSLOduration=1.389626449 podStartE2EDuration="1.389626449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:24.384066814 +0000 UTC m=+2245.967496520" watchObservedRunningTime="2026-10-30 13:23:24.389626449 +0000 UTC m=+2245.973056142"
Oct 30 13:23:24.766298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:24.765688    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:24.921256 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:24.921225    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:24.935309 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:24.935262    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:25.405710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:25.405679    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:23:27.323191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:27.323147    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg" podUID="cae94263-d227-4d32-9acd-bb4615ce48d3" containerName="payload-processing" containerID="cri-o://c34761c162709a163f997eda58b05c8c63fb52ff603f3e09830a7a71299e9445" gracePeriod=25
Oct 30 13:23:27.361513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:27.361431    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k" podUID="77464420-74b0-4fb5-a533-3ad3a79ff914" containerName="payload-pre-processing" containerID="cri-o://d53be3e3f9b4a9f3705a0b02bc584c81e492cf18284c29c8c438029a8c3c9b21" gracePeriod=25
Oct 30 13:23:33.751488 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:33.751446    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:33.764413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:33.764386    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:34.468317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:34.468267    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:23:36.822527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:36.822483    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg" podUID="cae94263-d227-4d32-9acd-bb4615ce48d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:23:36.822917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:36.822482    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k" podUID="77464420-74b0-4fb5-a533-3ad3a79ff914" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:23:38.206764 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.206721    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"]
Oct 30 13:23:38.211852 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.211828    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.240887 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.240857    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-60a9a6-openshift-default-dockercfg-f8sb4\""
Oct 30 13:23:38.246401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.246371    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"]
Oct 30 13:23:38.318985 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.318944    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-envoy\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.319162 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.319010    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-workload-certs\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.319162 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.319115    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-token\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.319162 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.319157    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-credential-socket\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.319306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.319211    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9a250d6-c25a-4160-8735-21bcb5a51f33-istiod-ca-cert\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.319306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.319250    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-data\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.319377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.319350    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-workload-socket\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.319414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.319377    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-podinfo\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.319592 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.319574    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8whx\" (UniqueName: \"kubernetes.io/projected/c9a250d6-c25a-4160-8735-21bcb5a51f33-kube-api-access-f8whx\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.420774 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.420740    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8whx\" (UniqueName: \"kubernetes.io/projected/c9a250d6-c25a-4160-8735-21bcb5a51f33-kube-api-access-f8whx\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.420951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.420825    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-envoy\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.420951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.420880    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-workload-certs\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.421043 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.420953    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-token\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.421043 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.421001    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-credential-socket\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.421145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.421054    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9a250d6-c25a-4160-8735-21bcb5a51f33-istiod-ca-cert\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.421145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.421097    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-data\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.421145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.421128    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-workload-socket\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.421343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.421165    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-podinfo\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.421487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.421457    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-credential-socket\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.421638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.421610    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-data\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.421774 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.421622    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-workload-socket\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.421774 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.421673    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-workload-certs\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.421869 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.421827    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9a250d6-c25a-4160-8735-21bcb5a51f33-istiod-ca-cert\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.423417 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.423399    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-envoy\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.423897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.423878    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-podinfo\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.444435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.444403    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-token\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.444574 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.444462    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8whx\" (UniqueName: \"kubernetes.io/projected/c9a250d6-c25a-4160-8735-21bcb5a51f33-kube-api-access-f8whx\") pod \"e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") " pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.526897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.526803    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:38.872395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.872366    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"]
Oct 30 13:23:38.872860 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:23:38.872833    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a250d6_c25a_4160_8735_21bcb5a51f33.slice/crio-1cff25e6dc67fb145a20576d12fbbe31d143c2b4641bc0e2759b5cbd543d674d WatchSource:0}: Error finding container 1cff25e6dc67fb145a20576d12fbbe31d143c2b4641bc0e2759b5cbd543d674d: Status 404 returned error can't find the container with id 1cff25e6dc67fb145a20576d12fbbe31d143c2b4641bc0e2759b5cbd543d674d
Oct 30 13:23:38.876469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.876429    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:23:38.876575 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.876499    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:23:38.876575 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:38.876531    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:23:39.503289 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:39.503244    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85" event={"ID":"c9a250d6-c25a-4160-8735-21bcb5a51f33","Type":"ContainerStarted","Data":"742c43ef645a6371794accbacea41bea87b8ec86c765b284d1d22d9dfa5b156b"}
Oct 30 13:23:39.503733 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:39.503316    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85" event={"ID":"c9a250d6-c25a-4160-8735-21bcb5a51f33","Type":"ContainerStarted","Data":"1cff25e6dc67fb145a20576d12fbbe31d143c2b4641bc0e2759b5cbd543d674d"}
Oct 30 13:23:39.527810 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:39.527780    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:39.529430 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:39.529406    5893 patch_prober.go:29] interesting pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:23:39.529553 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:39.529456    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85" podUID="c9a250d6-c25a-4160-8735-21bcb5a51f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 13:23:39.565002 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:39.564937    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85" podStartSLOduration=1.564917453 podStartE2EDuration="1.564917453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:39.560417079 +0000 UTC m=+2261.143846791" watchObservedRunningTime="2026-10-30 13:23:39.564917453 +0000 UTC m=+2261.148347147"
Oct 30 13:23:40.533295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:40.529890    5893 patch_prober.go:29] interesting pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:23:40.533295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:40.529947    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85" podUID="c9a250d6-c25a-4160-8735-21bcb5a51f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 13:23:41.527448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:41.527420    5893 patch_prober.go:29] interesting pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:23:41.527622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:41.527487    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85" podUID="c9a250d6-c25a-4160-8735-21bcb5a51f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 13:23:42.527220 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:42.527186    5893 patch_prober.go:29] interesting pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:23:42.527678 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:42.527243    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85" podUID="c9a250d6-c25a-4160-8735-21bcb5a51f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 13:23:43.527843 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:43.527794    5893 patch_prober.go:29] interesting pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:23:43.528432 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:43.528406    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85" podUID="c9a250d6-c25a-4160-8735-21bcb5a51f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 13:23:44.527141 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:44.527113    5893 patch_prober.go:29] interesting pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:23:44.527358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:44.527173    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85" podUID="c9a250d6-c25a-4160-8735-21bcb5a51f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 13:23:45.528201 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:45.528162    5893 patch_prober.go:29] interesting pod/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Oct 30 13:23:45.528622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:45.528233    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85" podUID="c9a250d6-c25a-4160-8735-21bcb5a51f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Oct 30 13:23:46.535288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:46.535244    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:46.535774 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:46.535523    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:46.536221 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:46.536203    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:23:46.822460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:46.822374    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k" podUID="77464420-74b0-4fb5-a533-3ad3a79ff914" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:23:46.822460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:46.822393    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg" podUID="cae94263-d227-4d32-9acd-bb4615ce48d3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:23:47.651959 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.651374    5893 generic.go:356] "Generic (PLEG): container finished" podID="cae94263-d227-4d32-9acd-bb4615ce48d3" containerID="c34761c162709a163f997eda58b05c8c63fb52ff603f3e09830a7a71299e9445" exitCode=0
Oct 30 13:23:47.651959 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.651545    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg" event={"ID":"cae94263-d227-4d32-9acd-bb4615ce48d3","Type":"ContainerDied","Data":"c34761c162709a163f997eda58b05c8c63fb52ff603f3e09830a7a71299e9445"}
Oct 30 13:23:47.654448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.654376    5893 generic.go:356] "Generic (PLEG): container finished" podID="77464420-74b0-4fb5-a533-3ad3a79ff914" containerID="d53be3e3f9b4a9f3705a0b02bc584c81e492cf18284c29c8c438029a8c3c9b21" exitCode=0
Oct 30 13:23:47.656093 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.656065    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k" event={"ID":"77464420-74b0-4fb5-a533-3ad3a79ff914","Type":"ContainerDied","Data":"d53be3e3f9b4a9f3705a0b02bc584c81e492cf18284c29c8c438029a8c3c9b21"}
Oct 30 13:23:47.716009 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.715977    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"
Oct 30 13:23:47.732605 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.732566    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77464420-74b0-4fb5-a533-3ad3a79ff914-kube-api-access-f59zr\" (UniqueName: \"kubernetes.io/projected/77464420-74b0-4fb5-a533-3ad3a79ff914-kube-api-access-f59zr\") pod \"77464420-74b0-4fb5-a533-3ad3a79ff914\" (UID: \"77464420-74b0-4fb5-a533-3ad3a79ff914\") "
Oct 30 13:23:47.733047 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.732979    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77464420-74b0-4fb5-a533-3ad3a79ff914-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77464420-74b0-4fb5-a533-3ad3a79ff914-plugins-config-volume\") pod \"77464420-74b0-4fb5-a533-3ad3a79ff914\" (UID: \"77464420-74b0-4fb5-a533-3ad3a79ff914\") "
Oct 30 13:23:47.734397 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.734377    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77464420-74b0-4fb5-a533-3ad3a79ff914-plugins-config-volume" pod "77464420-74b0-4fb5-a533-3ad3a79ff914" (UID: "77464420-74b0-4fb5-a533-3ad3a79ff914"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:23:47.736866 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.736849    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77464420-74b0-4fb5-a533-3ad3a79ff914-kube-api-access-f59zr" pod "77464420-74b0-4fb5-a533-3ad3a79ff914" (UID: "77464420-74b0-4fb5-a533-3ad3a79ff914"). InnerVolumeSpecName "kube-api-access-f59zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:47.760511 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.760491    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"
Oct 30 13:23:47.835731 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.835699    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae94263-d227-4d32-9acd-bb4615ce48d3-kube-api-access-cfr7r\" (UniqueName: \"kubernetes.io/projected/cae94263-d227-4d32-9acd-bb4615ce48d3-kube-api-access-cfr7r\") pod \"cae94263-d227-4d32-9acd-bb4615ce48d3\" (UID: \"cae94263-d227-4d32-9acd-bb4615ce48d3\") "
Oct 30 13:23:47.835989 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.835969    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cae94263-d227-4d32-9acd-bb4615ce48d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cae94263-d227-4d32-9acd-bb4615ce48d3-plugins-config-volume\") pod \"cae94263-d227-4d32-9acd-bb4615ce48d3\" (UID: \"cae94263-d227-4d32-9acd-bb4615ce48d3\") "
Oct 30 13:23:47.836385 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.836359    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae94263-d227-4d32-9acd-bb4615ce48d3-plugins-config-volume" pod "cae94263-d227-4d32-9acd-bb4615ce48d3" (UID: "cae94263-d227-4d32-9acd-bb4615ce48d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:23:47.837652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.837632    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cae94263-d227-4d32-9acd-bb4615ce48d3-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:23:47.837734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.837658    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f59zr\" (UniqueName: \"kubernetes.io/projected/77464420-74b0-4fb5-a533-3ad3a79ff914-kube-api-access-f59zr\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:23:47.837734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.837672    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77464420-74b0-4fb5-a533-3ad3a79ff914-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:23:47.839567 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.839498    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae94263-d227-4d32-9acd-bb4615ce48d3-kube-api-access-cfr7r" pod "cae94263-d227-4d32-9acd-bb4615ce48d3" (UID: "cae94263-d227-4d32-9acd-bb4615ce48d3"). InnerVolumeSpecName "kube-api-access-cfr7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:47.938907 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:47.938832    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfr7r\" (UniqueName: \"kubernetes.io/projected/cae94263-d227-4d32-9acd-bb4615ce48d3-kube-api-access-cfr7r\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:23:48.647167 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:23:48.647059    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:23:48.667953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:48.667920    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"
Oct 30 13:23:48.667953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:48.667928    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg" event={"ID":"cae94263-d227-4d32-9acd-bb4615ce48d3","Type":"ContainerDied","Data":"4d7564f670e1f2aef862fd2d5397e7021be68d7379310db50ccd2c0493383bc1"}
Oct 30 13:23:48.668536 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:48.667989    5893 scope.go:122] "RemoveContainer" containerID="c34761c162709a163f997eda58b05c8c63fb52ff603f3e09830a7a71299e9445"
Oct 30 13:23:48.669815 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:48.669787    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k" event={"ID":"77464420-74b0-4fb5-a533-3ad3a79ff914","Type":"ContainerDied","Data":"37460eb47cd81c559a15736d511a53e7b3aef59f3592b452bf5f2564981ecfe9"}
Oct 30 13:23:48.669942 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:48.669803    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"
Oct 30 13:23:48.684095 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:48.684074    5893 scope.go:122] "RemoveContainer" containerID="d53be3e3f9b4a9f3705a0b02bc584c81e492cf18284c29c8c438029a8c3c9b21"
Oct 30 13:23:48.723691 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:48.723660    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"]
Oct 30 13:23:48.736605 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:48.736576    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cd2d2e-78b4d7c54b-46rvg"]
Oct 30 13:23:48.766385 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:48.766356    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"]
Oct 30 13:23:48.771193 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:48.771169    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cd2d2e-675468c86c-2gl8k"]
Oct 30 13:23:50.534627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:50.534597    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77464420-74b0-4fb5-a533-3ad3a79ff914" path="/var/lib/kubelet/pods/77464420-74b0-4fb5-a533-3ad3a79ff914/volumes"
Oct 30 13:23:50.535097 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:50.535083    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cae94263-d227-4d32-9acd-bb4615ce48d3" path="/var/lib/kubelet/pods/cae94263-d227-4d32-9acd-bb4615ce48d3/volumes"
Oct 30 13:23:59.827821 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:59.827787    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"]
Oct 30 13:23:59.828402 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:23:59.827990    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g" podUID="14fbc94f-88db-47d5-9cfe-d59e8e0c9874" containerName="istio-proxy" containerID="cri-o://cff9e3e26710e9b3415ecfc8acda2ffa8d6fa8a18cdadadfa2aaa2b8fc962f59" gracePeriod=30
Oct 30 13:24:04.299447 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:04.299409    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-8982g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:24:04.299839 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:04.299485    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g" podUID="14fbc94f-88db-47d5-9cfe-d59e8e0c9874" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:24:05.132021 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.131994    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 13:24:05.160936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.160899    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-workload-certs\") pod \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") "
Oct 30 13:24:05.161143 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.160952    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-kube-api-access-lpgts\" (UniqueName: \"kubernetes.io/projected/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-kube-api-access-lpgts\") pod \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") "
Oct 30 13:24:05.161143 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.161008    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-data\") pod \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") "
Oct 30 13:24:05.161143 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.161134    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-credential-socket\") pod \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") "
Oct 30 13:24:05.161350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.161146    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-workload-certs" pod "14fbc94f-88db-47d5-9cfe-d59e8e0c9874" (UID: "14fbc94f-88db-47d5-9cfe-d59e8e0c9874"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.161350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.161227    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-workload-socket\") pod \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") "
Oct 30 13:24:05.161350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.161264    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istiod-ca-cert\") pod \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") "
Oct 30 13:24:05.161520 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.161376    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-data" pod "14fbc94f-88db-47d5-9cfe-d59e8e0c9874" (UID: "14fbc94f-88db-47d5-9cfe-d59e8e0c9874"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.161520 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.161416    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-envoy\") pod \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") "
Oct 30 13:24:05.161520 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.161482    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-token\" (UniqueName: \"kubernetes.io/projected/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-token\") pod \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") "
Oct 30 13:24:05.161714 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.161518    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-podinfo\") pod \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\" (UID: \"14fbc94f-88db-47d5-9cfe-d59e8e0c9874\") "
Oct 30 13:24:05.161714 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.161540    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-credential-socket" pod "14fbc94f-88db-47d5-9cfe-d59e8e0c9874" (UID: "14fbc94f-88db-47d5-9cfe-d59e8e0c9874"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.161813 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.161796    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istiod-ca-cert" pod "14fbc94f-88db-47d5-9cfe-d59e8e0c9874" (UID: "14fbc94f-88db-47d5-9cfe-d59e8e0c9874"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:05.162099 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.162076    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-workload-socket" pod "14fbc94f-88db-47d5-9cfe-d59e8e0c9874" (UID: "14fbc94f-88db-47d5-9cfe-d59e8e0c9874"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.162562 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.162458    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.162562 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.162482    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.162562 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.162495    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.162562 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.162509    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.162562 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.162522    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.164317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.163747    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-kube-api-access-lpgts" pod "14fbc94f-88db-47d5-9cfe-d59e8e0c9874" (UID: "14fbc94f-88db-47d5-9cfe-d59e8e0c9874"). InnerVolumeSpecName "kube-api-access-lpgts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:05.164986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.164959    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-podinfo" pod "14fbc94f-88db-47d5-9cfe-d59e8e0c9874" (UID: "14fbc94f-88db-47d5-9cfe-d59e8e0c9874"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:05.165801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.165737    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-envoy" pod "14fbc94f-88db-47d5-9cfe-d59e8e0c9874" (UID: "14fbc94f-88db-47d5-9cfe-d59e8e0c9874"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:05.166128 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.166102    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-token" pod "14fbc94f-88db-47d5-9cfe-d59e8e0c9874" (UID: "14fbc94f-88db-47d5-9cfe-d59e8e0c9874"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:05.264146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.264114    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.264146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.264141    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.264146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.264150    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.264440 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.264158    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpgts\" (UniqueName: \"kubernetes.io/projected/14fbc94f-88db-47d5-9cfe-d59e8e0c9874-kube-api-access-lpgts\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.777489 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.777455    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"]
Oct 30 13:24:05.813609 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.813570    5893 generic.go:356] "Generic (PLEG): container finished" podID="14fbc94f-88db-47d5-9cfe-d59e8e0c9874" containerID="cff9e3e26710e9b3415ecfc8acda2ffa8d6fa8a18cdadadfa2aaa2b8fc962f59" exitCode=0
Oct 30 13:24:05.813609 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.813609    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"
Oct 30 13:24:05.813882 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.813643    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g" event={"ID":"14fbc94f-88db-47d5-9cfe-d59e8e0c9874","Type":"ContainerDied","Data":"cff9e3e26710e9b3415ecfc8acda2ffa8d6fa8a18cdadadfa2aaa2b8fc962f59"}
Oct 30 13:24:05.813882 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.813704    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g" event={"ID":"14fbc94f-88db-47d5-9cfe-d59e8e0c9874","Type":"ContainerDied","Data":"365e18e8a519f9438c8acb9bf9b1aefdddc594a362f1b1534812fc4bb8d521f9"}
Oct 30 13:24:05.813882 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.813736    5893 scope.go:122] "RemoveContainer" containerID="cff9e3e26710e9b3415ecfc8acda2ffa8d6fa8a18cdadadfa2aaa2b8fc962f59"
Oct 30 13:24:05.830869 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.830847    5893 scope.go:122] "RemoveContainer" containerID="cff9e3e26710e9b3415ecfc8acda2ffa8d6fa8a18cdadadfa2aaa2b8fc962f59"
Oct 30 13:24:05.831743 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:24:05.831680    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cff9e3e26710e9b3415ecfc8acda2ffa8d6fa8a18cdadadfa2aaa2b8fc962f59\": container with ID starting with cff9e3e26710e9b3415ecfc8acda2ffa8d6fa8a18cdadadfa2aaa2b8fc962f59 not found: ID does not exist" containerID="cff9e3e26710e9b3415ecfc8acda2ffa8d6fa8a18cdadadfa2aaa2b8fc962f59"
Oct 30 13:24:05.831743 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.831715    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cff9e3e26710e9b3415ecfc8acda2ffa8d6fa8a18cdadadfa2aaa2b8fc962f59"} err="failed to get container status \"cff9e3e26710e9b3415ecfc8acda2ffa8d6fa8a18cdadadfa2aaa2b8fc962f59\": rpc error: code = NotFound desc = could not find container \"cff9e3e26710e9b3415ecfc8acda2ffa8d6fa8a18cdadadfa2aaa2b8fc962f59\": container with ID starting with cff9e3e26710e9b3415ecfc8acda2ffa8d6fa8a18cdadadfa2aaa2b8fc962f59 not found: ID does not exist"
Oct 30 13:24:05.928001 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.921599    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"]
Oct 30 13:24:05.945131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:05.945098    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-8982g"]
Oct 30 13:24:06.533310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:06.533263    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14fbc94f-88db-47d5-9cfe-d59e8e0c9874" path="/var/lib/kubelet/pods/14fbc94f-88db-47d5-9cfe-d59e8e0c9874/volumes"
Oct 30 13:24:10.583872 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:10.583833    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"]
Oct 30 13:24:10.584668 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:10.584638    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb" podUID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" containerName="istio-proxy" containerID="cri-o://87036c112675f046b03277fa563ee85cc72108f3014eec0f2b5bfe4d6ff61995" gracePeriod=30
Oct 30 13:24:12.002680 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:11.995203    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"]
Oct 30 13:24:12.004729 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.004698    5893 cpu_manager.go:414] "containerMap: removing container" podUID="14fbc94f-88db-47d5-9cfe-d59e8e0c9874" containerName="istio-proxy"
Oct 30 13:24:12.004950 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.004931    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14fbc94f-88db-47d5-9cfe-d59e8e0c9874" containerName="istio-proxy"
Oct 30 13:24:12.005107 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.005094    5893 cpu_manager.go:414] "containerMap: removing container" podUID="77464420-74b0-4fb5-a533-3ad3a79ff914" containerName="payload-pre-processing"
Oct 30 13:24:12.005201 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.005188    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77464420-74b0-4fb5-a533-3ad3a79ff914" containerName="payload-pre-processing"
Oct 30 13:24:12.005385 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.005371    5893 cpu_manager.go:414] "containerMap: removing container" podUID="cae94263-d227-4d32-9acd-bb4615ce48d3" containerName="payload-processing"
Oct 30 13:24:12.005486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.005474    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae94263-d227-4d32-9acd-bb4615ce48d3" containerName="payload-processing"
Oct 30 13:24:12.005768 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.005749    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="77464420-74b0-4fb5-a533-3ad3a79ff914" containerName="payload-pre-processing"
Oct 30 13:24:12.005907 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.005894    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae94263-d227-4d32-9acd-bb4615ce48d3" containerName="payload-processing"
Oct 30 13:24:12.006009 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.005998    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="14fbc94f-88db-47d5-9cfe-d59e8e0c9874" containerName="istio-proxy"
Oct 30 13:24:12.012321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.012304    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.035200 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.035164    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"]
Oct 30 13:24:12.047542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.047513    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-51288b-openshift-default-dockercfg-bbqtx\""
Oct 30 13:24:12.200937 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.200903    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-credential-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.201486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.201458    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-podinfo\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.201773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.201733    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-data\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.202005 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.201988    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-envoy\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.209047 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.209024    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-workload-certs\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.209318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.209298    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcnjd\" (UniqueName: \"kubernetes.io/projected/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-kube-api-access-bcnjd\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.209515 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.209498    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-token\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.209693 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.209677    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-workload-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.209858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.209842    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istiod-ca-cert\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.311454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.311363    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-envoy\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.311454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.311437    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-workload-certs\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.311713 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.311498    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcnjd\" (UniqueName: \"kubernetes.io/projected/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-kube-api-access-bcnjd\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.311713 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.311543    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-token\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.311713 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.311590    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-workload-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.311713 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.311631    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istiod-ca-cert\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.311961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.311720    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-credential-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.311961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.311904    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-podinfo\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.312077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.311981    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-data\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.312487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.312458    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-data\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.312757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.312733    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-workload-certs\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.312890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.312868    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-credential-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.312962 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.312919    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-workload-socket\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.313011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.312988    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istiod-ca-cert\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.315647 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.315614    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-podinfo\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.315818 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.315800    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-envoy\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.336557 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.336258    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcnjd\" (UniqueName: \"kubernetes.io/projected/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-kube-api-access-bcnjd\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.338151 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.338122    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-token\") pod \"e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") " pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.345376 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.345346    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:12.743510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.743474    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"]
Oct 30 13:24:12.782338 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:24:12.781020    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaffe0a0_8e1d_4bbf_a044_67a5f3af103a.slice/crio-75f6324ec94f88744dc0015ce04921bd499f3ff6750f4b94f28940cf61dc2989 WatchSource:0}: Error finding container 75f6324ec94f88744dc0015ce04921bd499f3ff6750f4b94f28940cf61dc2989: Status 404 returned error can't find the container with id 75f6324ec94f88744dc0015ce04921bd499f3ff6750f4b94f28940cf61dc2989
Oct 30 13:24:12.786031 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.785989    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:24:12.786598 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.786568    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:24:12.786804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.786783    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:24:12.882434 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:12.882347    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx" event={"ID":"daffe0a0-8e1d-4bbf-a044-67a5f3af103a","Type":"ContainerStarted","Data":"75f6324ec94f88744dc0015ce04921bd499f3ff6750f4b94f28940cf61dc2989"}
Oct 30 13:24:13.894108 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:13.894068    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx" event={"ID":"daffe0a0-8e1d-4bbf-a044-67a5f3af103a","Type":"ContainerStarted","Data":"3b51a2e3313b390ba71d9224ab277bbb3367033ee59b1e0646a69722d0debff5"}
Oct 30 13:24:13.976650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:13.976592    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx" podStartSLOduration=2.976575484 podStartE2EDuration="2.976575484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:13.973169506 +0000 UTC m=+2295.556599221" watchObservedRunningTime="2026-10-30 13:24:13.976575484 +0000 UTC m=+2295.560005177"
Oct 30 13:24:14.345914 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:14.345582    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:14.352451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:14.352425    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:14.901853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:14.901824    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:14.902845 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:14.902823    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:24:15.891899 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:15.891872    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:24:15.920085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:15.920045    5893 generic.go:356] "Generic (PLEG): container finished" podID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" containerID="87036c112675f046b03277fa563ee85cc72108f3014eec0f2b5bfe4d6ff61995" exitCode=0
Oct 30 13:24:15.920546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:15.920349    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"
Oct 30 13:24:15.920546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:15.920422    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb" event={"ID":"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611","Type":"ContainerDied","Data":"87036c112675f046b03277fa563ee85cc72108f3014eec0f2b5bfe4d6ff61995"}
Oct 30 13:24:15.920546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:15.920513    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb" event={"ID":"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611","Type":"ContainerDied","Data":"98070b0daaf6beadeee5db5fc9997446259534c0858c90f93ceb6b3b0d460610"}
Oct 30 13:24:15.920726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:15.920553    5893 scope.go:122] "RemoveContainer" containerID="87036c112675f046b03277fa563ee85cc72108f3014eec0f2b5bfe4d6ff61995"
Oct 30 13:24:15.934220 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:15.934197    5893 scope.go:122] "RemoveContainer" containerID="c36824bd757c6bdd5c0783ba26bc38c0f734b8a1017fac5f9053699202edd58c"
Oct 30 13:24:15.946956 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:15.946931    5893 scope.go:122] "RemoveContainer" containerID="87036c112675f046b03277fa563ee85cc72108f3014eec0f2b5bfe4d6ff61995"
Oct 30 13:24:15.947342 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:24:15.947317    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87036c112675f046b03277fa563ee85cc72108f3014eec0f2b5bfe4d6ff61995\": container with ID starting with 87036c112675f046b03277fa563ee85cc72108f3014eec0f2b5bfe4d6ff61995 not found: ID does not exist" containerID="87036c112675f046b03277fa563ee85cc72108f3014eec0f2b5bfe4d6ff61995"
Oct 30 13:24:15.947449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:15.947355    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87036c112675f046b03277fa563ee85cc72108f3014eec0f2b5bfe4d6ff61995"} err="failed to get container status \"87036c112675f046b03277fa563ee85cc72108f3014eec0f2b5bfe4d6ff61995\": rpc error: code = NotFound desc = could not find container \"87036c112675f046b03277fa563ee85cc72108f3014eec0f2b5bfe4d6ff61995\": container with ID starting with 87036c112675f046b03277fa563ee85cc72108f3014eec0f2b5bfe4d6ff61995 not found: ID does not exist"
Oct 30 13:24:15.947449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:15.947440    5893 scope.go:122] "RemoveContainer" containerID="c36824bd757c6bdd5c0783ba26bc38c0f734b8a1017fac5f9053699202edd58c"
Oct 30 13:24:15.948021 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:24:15.947993    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36824bd757c6bdd5c0783ba26bc38c0f734b8a1017fac5f9053699202edd58c\": container with ID starting with c36824bd757c6bdd5c0783ba26bc38c0f734b8a1017fac5f9053699202edd58c not found: ID does not exist" containerID="c36824bd757c6bdd5c0783ba26bc38c0f734b8a1017fac5f9053699202edd58c"
Oct 30 13:24:15.948121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:15.948030    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36824bd757c6bdd5c0783ba26bc38c0f734b8a1017fac5f9053699202edd58c"} err="failed to get container status \"c36824bd757c6bdd5c0783ba26bc38c0f734b8a1017fac5f9053699202edd58c\": rpc error: code = NotFound desc = could not find container \"c36824bd757c6bdd5c0783ba26bc38c0f734b8a1017fac5f9053699202edd58c\": container with ID starting with c36824bd757c6bdd5c0783ba26bc38c0f734b8a1017fac5f9053699202edd58c not found: ID does not exist"
Oct 30 13:24:16.022587 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.022556    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-kube-api-access-k496z\" (UniqueName: \"kubernetes.io/projected/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-kube-api-access-k496z\") pod \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") "
Oct 30 13:24:16.022800 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.022633    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-podinfo\") pod \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") "
Oct 30 13:24:16.022800 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.022693    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-credential-socket\") pod \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") "
Oct 30 13:24:16.022800 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.022788    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-data\") pod \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") "
Oct 30 13:24:16.022972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.022847    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-envoy\") pod \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") "
Oct 30 13:24:16.022972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.022928    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-workload-certs\") pod \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") "
Oct 30 13:24:16.023080 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.022969    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-workload-socket\") pod \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") "
Oct 30 13:24:16.023080 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.023023    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-token\" (UniqueName: \"kubernetes.io/projected/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-token\") pod \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") "
Oct 30 13:24:16.023080 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.023022    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-credential-socket" pod "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" (UID: "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:16.023080 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.023047    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-data" pod "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" (UID: "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:16.023336 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.023112    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istiod-ca-cert\") pod \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\" (UID: \"e8fd70b1-4864-4bcb-aba3-78a9a5ac2611\") "
Oct 30 13:24:16.023336 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.023208    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-workload-certs" pod "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" (UID: "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:16.023336 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.023236    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-workload-socket" pod "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" (UID: "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:16.023699 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.023601    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istiod-ca-cert" pod "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" (UID: "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:16.024323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.024295    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:16.024323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.024321    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:16.024486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.024334    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:16.024486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.024348    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:16.024486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.024363    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:16.025419 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.025395    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-kube-api-access-k496z" pod "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" (UID: "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611"). InnerVolumeSpecName "kube-api-access-k496z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:16.025544 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.025452    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-envoy" pod "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" (UID: "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:16.025605 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.025576    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-podinfo" pod "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" (UID: "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:16.025727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.025713    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-token" pod "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" (UID: "e8fd70b1-4864-4bcb-aba3-78a9a5ac2611"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:16.125992 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.125914    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:16.125992 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.125941    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k496z\" (UniqueName: \"kubernetes.io/projected/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-kube-api-access-k496z\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:16.125992 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.125952    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:16.125992 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.125961    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:16.268535 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.268502    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"]
Oct 30 13:24:16.275950 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.275925    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cd2d2e-openshift-default-57fd5565d9-tshdb"]
Oct 30 13:24:16.566217 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:16.566158    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" path="/var/lib/kubelet/pods/e8fd70b1-4864-4bcb-aba3-78a9a5ac2611/volumes"
Oct 30 13:24:18.670537 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:24:18.659761    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:24:20.809170 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:20.809134    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm" podUID="4356b5cd-aabc-41d4-89aa-d50586ce1952" containerName="main" containerID="cri-o://19bb7f26dcff56674d0f36edcb016bcdaa08d67fb7a599bc99d67b386944b76a" gracePeriod=45
Oct 30 13:24:20.974369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:20.974333    5893 generic.go:356] "Generic (PLEG): container finished" podID="4356b5cd-aabc-41d4-89aa-d50586ce1952" containerID="19bb7f26dcff56674d0f36edcb016bcdaa08d67fb7a599bc99d67b386944b76a" exitCode=0
Oct 30 13:24:20.974568 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:20.974402    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm" event={"ID":"4356b5cd-aabc-41d4-89aa-d50586ce1952","Type":"ContainerDied","Data":"19bb7f26dcff56674d0f36edcb016bcdaa08d67fb7a599bc99d67b386944b76a"}
Oct 30 13:24:21.125255 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.125229    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:24:21.203375 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.203346    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4356b5cd-aabc-41d4-89aa-d50586ce1952-tls-certs\" (UniqueName: \"kubernetes.io/secret/4356b5cd-aabc-41d4-89aa-d50586ce1952-tls-certs\") pod \"4356b5cd-aabc-41d4-89aa-d50586ce1952\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") "
Oct 30 13:24:21.203592 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.203480    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-home\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-home\") pod \"4356b5cd-aabc-41d4-89aa-d50586ce1952\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") "
Oct 30 13:24:21.203592 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.203562    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-model-cache\") pod \"4356b5cd-aabc-41d4-89aa-d50586ce1952\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") "
Oct 30 13:24:21.203726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.203619    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4356b5cd-aabc-41d4-89aa-d50586ce1952-kube-api-access-wgm6c\" (UniqueName: \"kubernetes.io/projected/4356b5cd-aabc-41d4-89aa-d50586ce1952-kube-api-access-wgm6c\") pod \"4356b5cd-aabc-41d4-89aa-d50586ce1952\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") "
Oct 30 13:24:21.203726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.203674    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-dshm\") pod \"4356b5cd-aabc-41d4-89aa-d50586ce1952\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") "
Oct 30 13:24:21.203837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.203770    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-tmp-dir\") pod \"4356b5cd-aabc-41d4-89aa-d50586ce1952\" (UID: \"4356b5cd-aabc-41d4-89aa-d50586ce1952\") "
Oct 30 13:24:21.203959 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.203931    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-home" pod "4356b5cd-aabc-41d4-89aa-d50586ce1952" (UID: "4356b5cd-aabc-41d4-89aa-d50586ce1952"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:21.204255 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.204229    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-model-cache" pod "4356b5cd-aabc-41d4-89aa-d50586ce1952" (UID: "4356b5cd-aabc-41d4-89aa-d50586ce1952"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:21.204526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.204502    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-tmp-dir" pod "4356b5cd-aabc-41d4-89aa-d50586ce1952" (UID: "4356b5cd-aabc-41d4-89aa-d50586ce1952"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:21.204679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.204660    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.204758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.204683    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.206966 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.206942    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4356b5cd-aabc-41d4-89aa-d50586ce1952-kube-api-access-wgm6c" pod "4356b5cd-aabc-41d4-89aa-d50586ce1952" (UID: "4356b5cd-aabc-41d4-89aa-d50586ce1952"). InnerVolumeSpecName "kube-api-access-wgm6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:21.206966 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.206951    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4356b5cd-aabc-41d4-89aa-d50586ce1952-tls-certs" pod "4356b5cd-aabc-41d4-89aa-d50586ce1952" (UID: "4356b5cd-aabc-41d4-89aa-d50586ce1952"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:24:21.207169 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.207137    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-dshm" pod "4356b5cd-aabc-41d4-89aa-d50586ce1952" (UID: "4356b5cd-aabc-41d4-89aa-d50586ce1952"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:21.306072 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.306037    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgm6c\" (UniqueName: \"kubernetes.io/projected/4356b5cd-aabc-41d4-89aa-d50586ce1952-kube-api-access-wgm6c\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.306072 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.306067    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.306072 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.306078    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4356b5cd-aabc-41d4-89aa-d50586ce1952-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.306333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.306088    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4356b5cd-aabc-41d4-89aa-d50586ce1952-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:21.983228 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.983194    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"
Oct 30 13:24:21.983228 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.983225    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm" event={"ID":"4356b5cd-aabc-41d4-89aa-d50586ce1952","Type":"ContainerDied","Data":"c6573e69d0e14195b7cc66322583e79cf94c9ee4e1e8b892853e614fdf1aa2b4"}
Oct 30 13:24:21.983822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:21.983302    5893 scope.go:122] "RemoveContainer" containerID="19bb7f26dcff56674d0f36edcb016bcdaa08d67fb7a599bc99d67b386944b76a"
Oct 30 13:24:22.003603 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:22.003575    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm" podUID="4356b5cd-aabc-41d4-89aa-d50586ce1952" containerName="main" probeResult="failure" output="Get \"https://10.134.0.139:8000/ready\": context deadline exceeded"
Oct 30 13:24:22.061305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:22.061260    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"]
Oct 30 13:24:22.073787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:22.073764    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-618c72/test-model-618c72-kserve-bb59f496f-wcxjm"]
Oct 30 13:24:22.533575 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:22.533544    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4356b5cd-aabc-41d4-89aa-d50586ce1952" path="/var/lib/kubelet/pods/4356b5cd-aabc-41d4-89aa-d50586ce1952/volumes"
Oct 30 13:24:30.563266 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.563228    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"]
Oct 30 13:24:30.564473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.564452    5893 cpu_manager.go:414] "containerMap: removing container" podUID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" containerName="istio-proxy"
Oct 30 13:24:30.564547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.564475    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" containerName="istio-proxy"
Oct 30 13:24:30.564547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.564511    5893 cpu_manager.go:414] "containerMap: removing container" podUID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" containerName="istio-proxy"
Oct 30 13:24:30.564547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.564521    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" containerName="istio-proxy"
Oct 30 13:24:30.564699 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.564551    5893 cpu_manager.go:414] "containerMap: removing container" podUID="4356b5cd-aabc-41d4-89aa-d50586ce1952" containerName="main"
Oct 30 13:24:30.564699 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.564557    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4356b5cd-aabc-41d4-89aa-d50586ce1952" containerName="main"
Oct 30 13:24:30.564805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.564722    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" containerName="istio-proxy"
Oct 30 13:24:30.564805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.564739    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8fd70b1-4864-4bcb-aba3-78a9a5ac2611" containerName="istio-proxy"
Oct 30 13:24:30.564805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.564754    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="4356b5cd-aabc-41d4-89aa-d50586ce1952" containerName="main"
Oct 30 13:24:30.569137 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.569114    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.590463 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.590432    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"]
Oct 30 13:24:30.597986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.597959    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-acd125-openshift-default-dockercfg-kct5r\""
Oct 30 13:24:30.647992 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.647953    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-envoy\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.648191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.648056    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-workload-certs\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.648191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.648089    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-data\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.648327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.648189    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istiod-ca-cert\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.648365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.648326    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-token\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.648495 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.648474    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-workload-socket\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.648576 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.648556    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-podinfo\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.648701 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.648687    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-credential-socket\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.648758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.648715    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsjkp\" (UniqueName: \"kubernetes.io/projected/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-kube-api-access-wsjkp\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.750008 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.749971    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-workload-socket\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.750196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.750038    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-podinfo\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.750196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.750142    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-credential-socket\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.750196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.750167    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsjkp\" (UniqueName: \"kubernetes.io/projected/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-kube-api-access-wsjkp\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.750379 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.750365    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-envoy\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.750434 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.750411    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-workload-certs\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.751078 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.750510    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-credential-socket\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.751078 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.750553    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-data\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.751078 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.750704    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istiod-ca-cert\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.751078 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.750769    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-token\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.751078 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.750780    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-data\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.751078 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.751016    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-workload-certs\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.751503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.751113    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-workload-socket\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.751563 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.751520    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istiod-ca-cert\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.753300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.753248    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-podinfo\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.753555 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.753530    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-envoy\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.778743 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.778709    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsjkp\" (UniqueName: \"kubernetes.io/projected/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-kube-api-access-wsjkp\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.779291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.779232    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-token\") pod \"e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") " pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.784175 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.784153    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"]
Oct 30 13:24:30.789954 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.789937    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:30.810077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.809763    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a5e47c-openshift-default-dockercfg-5hq7k\""
Oct 30 13:24:30.813792 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.813740    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"]
Oct 30 13:24:30.889087 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.889047    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:30.956320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.953896    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-credential-socket\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:30.956320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.953950    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb104279-c853-4843-a0f7-e55eb170c620-istiod-ca-cert\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:30.956320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.954086    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-workload-socket\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:30.956320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.954168    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb104279-c853-4843-a0f7-e55eb170c620-istio-token\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:30.956320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.954203    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-istio-envoy\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:30.956320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.954230    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68x7g\" (UniqueName: \"kubernetes.io/projected/fb104279-c853-4843-a0f7-e55eb170c620-kube-api-access-68x7g\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:30.956320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.954334    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-istio-data\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:30.956320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.954465    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-workload-certs\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:30.956320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:30.954544    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb104279-c853-4843-a0f7-e55eb170c620-istio-podinfo\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.058650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.058608    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb104279-c853-4843-a0f7-e55eb170c620-istiod-ca-cert\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.058836 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.058710    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-workload-socket\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.058836 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.058791    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb104279-c853-4843-a0f7-e55eb170c620-istio-token\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.058836 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.058821    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-istio-envoy\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.059001 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.058850    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68x7g\" (UniqueName: \"kubernetes.io/projected/fb104279-c853-4843-a0f7-e55eb170c620-kube-api-access-68x7g\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.059001 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.058923    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-istio-data\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.059102 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.059073    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-workload-certs\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.059173 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.059158    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb104279-c853-4843-a0f7-e55eb170c620-istio-podinfo\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.059317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.059301    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-credential-socket\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.060692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.059784    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-credential-socket\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.060692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.059808    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb104279-c853-4843-a0f7-e55eb170c620-istiod-ca-cert\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.060692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.060131    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-workload-socket\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.060692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.060429    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-istio-data\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.060692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.060650    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-workload-certs\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.063374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.063313    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-istio-envoy\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.065038 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.064987    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb104279-c853-4843-a0f7-e55eb170c620-istio-podinfo\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.087964 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.087908    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68x7g\" (UniqueName: \"kubernetes.io/projected/fb104279-c853-4843-a0f7-e55eb170c620-kube-api-access-68x7g\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.095087 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.095061    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb104279-c853-4843-a0f7-e55eb170c620-istio-token\") pod \"e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") " pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.110910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.110884    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:31.224154 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.224099    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"]
Oct 30 13:24:31.229784 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:24:31.229752    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fb1fae_c51f_4f04_a6ae_110f42dfd9eb.slice/crio-5b8e245f857da7d3829135a995717c0125dc5ebcda8bb2be78c39916e20c750b WatchSource:0}: Error finding container 5b8e245f857da7d3829135a995717c0125dc5ebcda8bb2be78c39916e20c750b: Status 404 returned error can't find the container with id 5b8e245f857da7d3829135a995717c0125dc5ebcda8bb2be78c39916e20c750b
Oct 30 13:24:31.233616 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.233576    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:24:31.233697 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.233672    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:24:31.233750 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.233724    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:24:31.439300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.431835    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"]
Oct 30 13:24:31.441954 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:24:31.441314    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb104279_c853_4843_a0f7_e55eb170c620.slice/crio-b04078a1261f9b42895a89a0329d7eaead77fed59bfd0144eba69258d9d398f9 WatchSource:0}: Error finding container b04078a1261f9b42895a89a0329d7eaead77fed59bfd0144eba69258d9d398f9: Status 404 returned error can't find the container with id b04078a1261f9b42895a89a0329d7eaead77fed59bfd0144eba69258d9d398f9
Oct 30 13:24:31.446030 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.445978    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:24:31.446248 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.446227    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:24:31.446391 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:31.446376    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:24:32.074296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:32.074069    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44" event={"ID":"fb104279-c853-4843-a0f7-e55eb170c620","Type":"ContainerStarted","Data":"255da8addcadf2572c0ad665014a5163d16ee9a523a99044e79362d90510fdb8"}
Oct 30 13:24:32.074296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:32.074150    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44" event={"ID":"fb104279-c853-4843-a0f7-e55eb170c620","Type":"ContainerStarted","Data":"b04078a1261f9b42895a89a0329d7eaead77fed59bfd0144eba69258d9d398f9"}
Oct 30 13:24:32.076191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:32.076160    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5" event={"ID":"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb","Type":"ContainerStarted","Data":"8c1ba04c65ab34c2495913a5ec3b070904f7eaacbe28b282d22cbe0f6d076f09"}
Oct 30 13:24:32.076390 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:32.076213    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5" event={"ID":"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb","Type":"ContainerStarted","Data":"5b8e245f857da7d3829135a995717c0125dc5ebcda8bb2be78c39916e20c750b"}
Oct 30 13:24:32.111658 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:32.111624    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:32.113351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:32.113323    5893 patch_prober.go:29] interesting pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 13:24:32.113485 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:32.113381    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44" podUID="fb104279-c853-4843-a0f7-e55eb170c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 13:24:32.121800 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:32.121755    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44" podStartSLOduration=2.121742415 podStartE2EDuration="2.121742415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:32.121319997 +0000 UTC m=+2313.704749684" watchObservedRunningTime="2026-10-30 13:24:32.121742415 +0000 UTC m=+2313.705172107"
Oct 30 13:24:32.174656 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:32.174573    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5" podStartSLOduration=2.1745541680000002 podStartE2EDuration="2.174554168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:32.174540883 +0000 UTC m=+2313.757970575" watchObservedRunningTime="2026-10-30 13:24:32.174554168 +0000 UTC m=+2313.757983871"
Oct 30 13:24:32.895200 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:32.893714    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:32.901335 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:32.900405    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 13:24:32.901335 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:32.900468    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5" podUID="54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 13:24:33.111355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:33.111316    5893 patch_prober.go:29] interesting pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 13:24:33.111886 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:33.111370    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44" podUID="fb104279-c853-4843-a0f7-e55eb170c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 13:24:33.895298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:33.894643    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 13:24:33.895298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:33.894704    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5" podUID="54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 13:24:34.113344 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:34.111802    5893 patch_prober.go:29] interesting pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 13:24:34.113344 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:34.111885    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44" podUID="fb104279-c853-4843-a0f7-e55eb170c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 13:24:34.930299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:34.926612    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 13:24:34.930299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:34.926672    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5" podUID="54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 13:24:35.111572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:35.111538    5893 patch_prober.go:29] interesting pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 13:24:35.111883 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:35.111835    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44" podUID="fb104279-c853-4843-a0f7-e55eb170c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 13:24:35.889742 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:35.889703    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 13:24:35.890441 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:35.890394    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5" podUID="54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 13:24:36.111912 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:36.111867    5893 patch_prober.go:29] interesting pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 13:24:36.112112 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:36.111925    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44" podUID="fb104279-c853-4843-a0f7-e55eb170c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 13:24:36.889728 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:36.889691    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 13:24:36.890040 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:36.890012    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5" podUID="54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 13:24:37.111523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:37.111490    5893 patch_prober.go:29] interesting pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 13:24:37.111734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:37.111715    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44" podUID="fb104279-c853-4843-a0f7-e55eb170c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 13:24:37.896230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:37.896192    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body=
Oct 30 13:24:37.897076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:37.897046    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5" podUID="54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused"
Oct 30 13:24:38.112256 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:38.112221    5893 patch_prober.go:29] interesting pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Oct 30 13:24:38.112527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:38.112496    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44" podUID="fb104279-c853-4843-a0f7-e55eb170c620" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Oct 30 13:24:38.895188 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:38.895134    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:38.895670 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:38.895652    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:38.896337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:38.896319    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:24:39.117083 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:39.117055    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:39.119907 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:39.119881    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:39.122683 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:39.122650    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:24:40.246790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:40.246749    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"]
Oct 30 13:24:40.281134 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:40.281103    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"]
Oct 30 13:24:40.553056 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:40.552967    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"]
Oct 30 13:24:40.553246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:40.553219    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5" podUID="379ef92e-3861-4eec-8eb2-c52787142d1a" containerName="istio-proxy" containerID="cri-o://3705a4f895d3d183b469bc76933be5bc4ecfbd490714362263411807ab8a0d6e" gracePeriod=30
Oct 30 13:24:45.247745 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.247666    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb" podUID="a3e89b62-a248-46fc-8e83-fc2d4fa66a02" containerName="payload-processing" containerID="cri-o://a40abb11c813733c53c9e99ff4267ed6acb82546b0f8c25b6e9ec6f1d041cce8" gracePeriod=25
Oct 30 13:24:45.282434 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.282398    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq" podUID="a814d069-870d-4ea0-945f-2ddcd1d517eb" containerName="payload-pre-processing" containerID="cri-o://6bd17ded77f7b16d969d5d0fd4c01ca24eb89dbaa2400456116506fbefc243df" gracePeriod=25
Oct 30 13:24:45.853508 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.853482    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:24:45.942586 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.942556    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-token\" (UniqueName: \"kubernetes.io/projected/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-token\") pod \"379ef92e-3861-4eec-8eb2-c52787142d1a\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") "
Oct 30 13:24:45.942788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.942613    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-podinfo\") pod \"379ef92e-3861-4eec-8eb2-c52787142d1a\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") "
Oct 30 13:24:45.942788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.942669    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-data\") pod \"379ef92e-3861-4eec-8eb2-c52787142d1a\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") "
Oct 30 13:24:45.942788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.942735    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/379ef92e-3861-4eec-8eb2-c52787142d1a-kube-api-access-26pb4\" (UniqueName: \"kubernetes.io/projected/379ef92e-3861-4eec-8eb2-c52787142d1a-kube-api-access-26pb4\") pod \"379ef92e-3861-4eec-8eb2-c52787142d1a\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") "
Oct 30 13:24:45.942969 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.942787    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-workload-socket\") pod \"379ef92e-3861-4eec-8eb2-c52787142d1a\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") "
Oct 30 13:24:45.942969 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.942860    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-envoy\") pod \"379ef92e-3861-4eec-8eb2-c52787142d1a\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") "
Oct 30 13:24:45.942969 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.942916    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-workload-certs\") pod \"379ef92e-3861-4eec-8eb2-c52787142d1a\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") "
Oct 30 13:24:45.943141 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.943126    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-credential-socket\") pod \"379ef92e-3861-4eec-8eb2-c52787142d1a\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") "
Oct 30 13:24:45.944443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.944420    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/379ef92e-3861-4eec-8eb2-c52787142d1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379ef92e-3861-4eec-8eb2-c52787142d1a-istiod-ca-cert\") pod \"379ef92e-3861-4eec-8eb2-c52787142d1a\" (UID: \"379ef92e-3861-4eec-8eb2-c52787142d1a\") "
Oct 30 13:24:45.945073 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.943764    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-workload-certs" pod "379ef92e-3861-4eec-8eb2-c52787142d1a" (UID: "379ef92e-3861-4eec-8eb2-c52787142d1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:45.945188 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.943803    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-workload-socket" pod "379ef92e-3861-4eec-8eb2-c52787142d1a" (UID: "379ef92e-3861-4eec-8eb2-c52787142d1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:45.945188 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.944006    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-credential-socket" pod "379ef92e-3861-4eec-8eb2-c52787142d1a" (UID: "379ef92e-3861-4eec-8eb2-c52787142d1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:45.945188 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.944020    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-data" pod "379ef92e-3861-4eec-8eb2-c52787142d1a" (UID: "379ef92e-3861-4eec-8eb2-c52787142d1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:45.945188 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.945030    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/379ef92e-3861-4eec-8eb2-c52787142d1a-istiod-ca-cert" pod "379ef92e-3861-4eec-8eb2-c52787142d1a" (UID: "379ef92e-3861-4eec-8eb2-c52787142d1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:45.946152 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.946048    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.946152 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.946075    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.946152 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.946091    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.946152 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.946108    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.946152 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.946123    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/379ef92e-3861-4eec-8eb2-c52787142d1a-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:45.947089 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.947018    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379ef92e-3861-4eec-8eb2-c52787142d1a-kube-api-access-26pb4" pod "379ef92e-3861-4eec-8eb2-c52787142d1a" (UID: "379ef92e-3861-4eec-8eb2-c52787142d1a"). InnerVolumeSpecName "kube-api-access-26pb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:45.948841 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.948814    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-token" pod "379ef92e-3861-4eec-8eb2-c52787142d1a" (UID: "379ef92e-3861-4eec-8eb2-c52787142d1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:45.949418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.949392    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-envoy" pod "379ef92e-3861-4eec-8eb2-c52787142d1a" (UID: "379ef92e-3861-4eec-8eb2-c52787142d1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:24:45.949595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:45.949562    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-podinfo" pod "379ef92e-3861-4eec-8eb2-c52787142d1a" (UID: "379ef92e-3861-4eec-8eb2-c52787142d1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:24:46.064706 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:46.064607    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:46.064706 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:46.064640    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:46.064706 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:46.064654    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26pb4\" (UniqueName: \"kubernetes.io/projected/379ef92e-3861-4eec-8eb2-c52787142d1a-kube-api-access-26pb4\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:46.064706 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:46.064666    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/379ef92e-3861-4eec-8eb2-c52787142d1a-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:46.224544 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:46.224510    5893 generic.go:356] "Generic (PLEG): container finished" podID="379ef92e-3861-4eec-8eb2-c52787142d1a" containerID="3705a4f895d3d183b469bc76933be5bc4ecfbd490714362263411807ab8a0d6e" exitCode=0
Oct 30 13:24:46.224734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:46.224562    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"
Oct 30 13:24:46.224734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:46.224595    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5" event={"ID":"379ef92e-3861-4eec-8eb2-c52787142d1a","Type":"ContainerDied","Data":"3705a4f895d3d183b469bc76933be5bc4ecfbd490714362263411807ab8a0d6e"}
Oct 30 13:24:46.224734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:46.224653    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5" event={"ID":"379ef92e-3861-4eec-8eb2-c52787142d1a","Type":"ContainerDied","Data":"8536e1bba7a929dd31f0e81e5d2fd311f9703114c1882a2c42e06e393dec96b5"}
Oct 30 13:24:46.224734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:46.224687    5893 scope.go:122] "RemoveContainer" containerID="3705a4f895d3d183b469bc76933be5bc4ecfbd490714362263411807ab8a0d6e"
Oct 30 13:24:46.250790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:46.250603    5893 scope.go:122] "RemoveContainer" containerID="3705a4f895d3d183b469bc76933be5bc4ecfbd490714362263411807ab8a0d6e"
Oct 30 13:24:46.251070 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:24:46.251002    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3705a4f895d3d183b469bc76933be5bc4ecfbd490714362263411807ab8a0d6e\": container with ID starting with 3705a4f895d3d183b469bc76933be5bc4ecfbd490714362263411807ab8a0d6e not found: ID does not exist" containerID="3705a4f895d3d183b469bc76933be5bc4ecfbd490714362263411807ab8a0d6e"
Oct 30 13:24:46.251070 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:46.251034    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3705a4f895d3d183b469bc76933be5bc4ecfbd490714362263411807ab8a0d6e"} err="failed to get container status \"3705a4f895d3d183b469bc76933be5bc4ecfbd490714362263411807ab8a0d6e\": rpc error: code = NotFound desc = could not find container \"3705a4f895d3d183b469bc76933be5bc4ecfbd490714362263411807ab8a0d6e\": container with ID starting with 3705a4f895d3d183b469bc76933be5bc4ecfbd490714362263411807ab8a0d6e not found: ID does not exist"
Oct 30 13:24:46.288461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:46.288424    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"]
Oct 30 13:24:46.300074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:46.300041    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-80fbd9-openshift-default-cf4656f6f-8n7z5"]
Oct 30 13:24:46.535059 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:46.534979    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="379ef92e-3861-4eec-8eb2-c52787142d1a" path="/var/lib/kubelet/pods/379ef92e-3861-4eec-8eb2-c52787142d1a/volumes"
Oct 30 13:24:47.006804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:47.006762    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq" podUID="a814d069-870d-4ea0-945f-2ddcd1d517eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:24:47.007225 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:47.007178    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb" podUID="a3e89b62-a248-46fc-8e83-fc2d4fa66a02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:24:48.647999 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:24:48.647965    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:24:51.602635 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:51.602598    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"]
Oct 30 13:24:51.639297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:51.639242    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"]
Oct 30 13:24:56.603794 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:56.603732    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj" podUID="967ac404-7b3d-49bd-9cb5-63e148bb4f9a" containerName="payload-processing" containerID="cri-o://6feebe38f4970f02657ce79c2667643f59944006a033407e7e61087a2bbdf596" gracePeriod=25
Oct 30 13:24:56.640557 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:56.640482    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr" podUID="0db246df-9cb2-4c31-9389-13fb0c8ac395" containerName="payload-pre-processing" containerID="cri-o://16747b4a97b604f70df517f77ff7bb6f9d51ab478fa89e49d34cd29cf6b4da1a" gracePeriod=25
Oct 30 13:24:56.907076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:56.907049    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"
Oct 30 13:24:56.968946 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:56.968909    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967ac404-7b3d-49bd-9cb5-63e148bb4f9a-kube-api-access-4vwj7\" (UniqueName: \"kubernetes.io/projected/967ac404-7b3d-49bd-9cb5-63e148bb4f9a-kube-api-access-4vwj7\") pod \"967ac404-7b3d-49bd-9cb5-63e148bb4f9a\" (UID: \"967ac404-7b3d-49bd-9cb5-63e148bb4f9a\") "
Oct 30 13:24:56.969342 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:56.969316    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967ac404-7b3d-49bd-9cb5-63e148bb4f9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967ac404-7b3d-49bd-9cb5-63e148bb4f9a-plugins-config-volume\") pod \"967ac404-7b3d-49bd-9cb5-63e148bb4f9a\" (UID: \"967ac404-7b3d-49bd-9cb5-63e148bb4f9a\") "
Oct 30 13:24:56.970435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:56.970409    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967ac404-7b3d-49bd-9cb5-63e148bb4f9a-plugins-config-volume" pod "967ac404-7b3d-49bd-9cb5-63e148bb4f9a" (UID: "967ac404-7b3d-49bd-9cb5-63e148bb4f9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:56.973166 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:56.973127    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967ac404-7b3d-49bd-9cb5-63e148bb4f9a-kube-api-access-4vwj7" pod "967ac404-7b3d-49bd-9cb5-63e148bb4f9a" (UID: "967ac404-7b3d-49bd-9cb5-63e148bb4f9a"). InnerVolumeSpecName "kube-api-access-4vwj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:57.005349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:57.005314    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb" podUID="a3e89b62-a248-46fc-8e83-fc2d4fa66a02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:24:57.005476 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:57.005432    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq" podUID="a814d069-870d-4ea0-945f-2ddcd1d517eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:24:57.071203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:57.071168    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/967ac404-7b3d-49bd-9cb5-63e148bb4f9a-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:57.071203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:57.071196    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vwj7\" (UniqueName: \"kubernetes.io/projected/967ac404-7b3d-49bd-9cb5-63e148bb4f9a-kube-api-access-4vwj7\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:24:57.317110 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:57.317076    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr" podUID="0db246df-9cb2-4c31-9389-13fb0c8ac395" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:24:57.372339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:57.372307    5893 generic.go:356] "Generic (PLEG): container finished" podID="967ac404-7b3d-49bd-9cb5-63e148bb4f9a" containerID="6feebe38f4970f02657ce79c2667643f59944006a033407e7e61087a2bbdf596" exitCode=0
Oct 30 13:24:57.372528 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:57.372427    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj" event={"ID":"967ac404-7b3d-49bd-9cb5-63e148bb4f9a","Type":"ContainerDied","Data":"6feebe38f4970f02657ce79c2667643f59944006a033407e7e61087a2bbdf596"}
Oct 30 13:24:57.372528 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:57.372481    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj" event={"ID":"967ac404-7b3d-49bd-9cb5-63e148bb4f9a","Type":"ContainerDied","Data":"c94eeb1e37b86344996642877d587dd5c0e8dd6ea7169c8678efccd60619ac1c"}
Oct 30 13:24:57.372528 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:57.372510    5893 scope.go:122] "RemoveContainer" containerID="6feebe38f4970f02657ce79c2667643f59944006a033407e7e61087a2bbdf596"
Oct 30 13:24:57.372833 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:57.372813    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"
Oct 30 13:24:57.385005 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:57.384983    5893 scope.go:122] "RemoveContainer" containerID="6feebe38f4970f02657ce79c2667643f59944006a033407e7e61087a2bbdf596"
Oct 30 13:24:57.385370 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:24:57.385338    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6feebe38f4970f02657ce79c2667643f59944006a033407e7e61087a2bbdf596\": container with ID starting with 6feebe38f4970f02657ce79c2667643f59944006a033407e7e61087a2bbdf596 not found: ID does not exist" containerID="6feebe38f4970f02657ce79c2667643f59944006a033407e7e61087a2bbdf596"
Oct 30 13:24:57.385474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:57.385392    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6feebe38f4970f02657ce79c2667643f59944006a033407e7e61087a2bbdf596"} err="failed to get container status \"6feebe38f4970f02657ce79c2667643f59944006a033407e7e61087a2bbdf596\": rpc error: code = NotFound desc = could not find container \"6feebe38f4970f02657ce79c2667643f59944006a033407e7e61087a2bbdf596\": container with ID starting with 6feebe38f4970f02657ce79c2667643f59944006a033407e7e61087a2bbdf596 not found: ID does not exist"
Oct 30 13:24:57.425691 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:57.425667    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"]
Oct 30 13:24:57.437098 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:57.437070    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bec983-7d8dddc58-tvcvj"]
Oct 30 13:24:58.533485 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:24:58.533448    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967ac404-7b3d-49bd-9cb5-63e148bb4f9a" path="/var/lib/kubelet/pods/967ac404-7b3d-49bd-9cb5-63e148bb4f9a/volumes"
Oct 30 13:25:05.471293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.470013    5893 generic.go:356] "Generic (PLEG): container finished" podID="a814d069-870d-4ea0-945f-2ddcd1d517eb" containerID="6bd17ded77f7b16d969d5d0fd4c01ca24eb89dbaa2400456116506fbefc243df" exitCode=0
Oct 30 13:25:05.471293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.470664    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq" event={"ID":"a814d069-870d-4ea0-945f-2ddcd1d517eb","Type":"ContainerDied","Data":"6bd17ded77f7b16d969d5d0fd4c01ca24eb89dbaa2400456116506fbefc243df"}
Oct 30 13:25:05.473461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.473346    5893 generic.go:356] "Generic (PLEG): container finished" podID="a3e89b62-a248-46fc-8e83-fc2d4fa66a02" containerID="a40abb11c813733c53c9e99ff4267ed6acb82546b0f8c25b6e9ec6f1d041cce8" exitCode=0
Oct 30 13:25:05.473805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.473785    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb" event={"ID":"a3e89b62-a248-46fc-8e83-fc2d4fa66a02","Type":"ContainerDied","Data":"a40abb11c813733c53c9e99ff4267ed6acb82546b0f8c25b6e9ec6f1d041cce8"}
Oct 30 13:25:05.633177 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.633152    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"
Oct 30 13:25:05.636077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.636056    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"
Oct 30 13:25:05.740973 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.740941    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3e89b62-a248-46fc-8e83-fc2d4fa66a02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3e89b62-a248-46fc-8e83-fc2d4fa66a02-plugins-config-volume\") pod \"a3e89b62-a248-46fc-8e83-fc2d4fa66a02\" (UID: \"a3e89b62-a248-46fc-8e83-fc2d4fa66a02\") "
Oct 30 13:25:05.741173 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.741027    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3e89b62-a248-46fc-8e83-fc2d4fa66a02-kube-api-access-rgbs5\" (UniqueName: \"kubernetes.io/projected/a3e89b62-a248-46fc-8e83-fc2d4fa66a02-kube-api-access-rgbs5\") pod \"a3e89b62-a248-46fc-8e83-fc2d4fa66a02\" (UID: \"a3e89b62-a248-46fc-8e83-fc2d4fa66a02\") "
Oct 30 13:25:05.741173 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.741074    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a814d069-870d-4ea0-945f-2ddcd1d517eb-kube-api-access-mj48b\" (UniqueName: \"kubernetes.io/projected/a814d069-870d-4ea0-945f-2ddcd1d517eb-kube-api-access-mj48b\") pod \"a814d069-870d-4ea0-945f-2ddcd1d517eb\" (UID: \"a814d069-870d-4ea0-945f-2ddcd1d517eb\") "
Oct 30 13:25:05.741173 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.741155    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a814d069-870d-4ea0-945f-2ddcd1d517eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a814d069-870d-4ea0-945f-2ddcd1d517eb-plugins-config-volume\") pod \"a814d069-870d-4ea0-945f-2ddcd1d517eb\" (UID: \"a814d069-870d-4ea0-945f-2ddcd1d517eb\") "
Oct 30 13:25:05.741423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.741400    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3e89b62-a248-46fc-8e83-fc2d4fa66a02-plugins-config-volume" pod "a3e89b62-a248-46fc-8e83-fc2d4fa66a02" (UID: "a3e89b62-a248-46fc-8e83-fc2d4fa66a02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:05.741843 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.741817    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a814d069-870d-4ea0-945f-2ddcd1d517eb-plugins-config-volume" pod "a814d069-870d-4ea0-945f-2ddcd1d517eb" (UID: "a814d069-870d-4ea0-945f-2ddcd1d517eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:05.742189 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.742170    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3e89b62-a248-46fc-8e83-fc2d4fa66a02-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:05.742265 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.742194    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a814d069-870d-4ea0-945f-2ddcd1d517eb-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:05.744220 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.744192    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e89b62-a248-46fc-8e83-fc2d4fa66a02-kube-api-access-rgbs5" pod "a3e89b62-a248-46fc-8e83-fc2d4fa66a02" (UID: "a3e89b62-a248-46fc-8e83-fc2d4fa66a02"). InnerVolumeSpecName "kube-api-access-rgbs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:05.744429 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.744216    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a814d069-870d-4ea0-945f-2ddcd1d517eb-kube-api-access-mj48b" pod "a814d069-870d-4ea0-945f-2ddcd1d517eb" (UID: "a814d069-870d-4ea0-945f-2ddcd1d517eb"). InnerVolumeSpecName "kube-api-access-mj48b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:05.844201 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.844123    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgbs5\" (UniqueName: \"kubernetes.io/projected/a3e89b62-a248-46fc-8e83-fc2d4fa66a02-kube-api-access-rgbs5\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:05.844201 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:05.844150    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj48b\" (UniqueName: \"kubernetes.io/projected/a814d069-870d-4ea0-945f-2ddcd1d517eb-kube-api-access-mj48b\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:06.482712 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:06.482677    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"
Oct 30 13:25:06.483202 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:06.482715    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq" event={"ID":"a814d069-870d-4ea0-945f-2ddcd1d517eb","Type":"ContainerDied","Data":"088c7183015c9331b2c88d0a09fcf2a96bee5716e3724ca9ae058b62acba610e"}
Oct 30 13:25:06.483202 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:06.482772    5893 scope.go:122] "RemoveContainer" containerID="6bd17ded77f7b16d969d5d0fd4c01ca24eb89dbaa2400456116506fbefc243df"
Oct 30 13:25:06.484761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:06.484735    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb" event={"ID":"a3e89b62-a248-46fc-8e83-fc2d4fa66a02","Type":"ContainerDied","Data":"bb02e8ff5add8c72694287971b7e1e2e6d9764c0f8ff34d8dfa1423a7c992275"}
Oct 30 13:25:06.484761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:06.484759    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"
Oct 30 13:25:06.499523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:06.498141    5893 scope.go:122] "RemoveContainer" containerID="a40abb11c813733c53c9e99ff4267ed6acb82546b0f8c25b6e9ec6f1d041cce8"
Oct 30 13:25:06.551258 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:06.551197    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"]
Oct 30 13:25:06.563573 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:06.563548    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-618c72-fb55d8bc5-gmhjq"]
Oct 30 13:25:06.593116 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:06.593090    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"]
Oct 30 13:25:06.596720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:06.596700    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-618c72-5bbdd8858-dnjxb"]
Oct 30 13:25:07.317039 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:07.316996    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr" podUID="0db246df-9cb2-4c31-9389-13fb0c8ac395" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:25:08.534255 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:08.534165    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3e89b62-a248-46fc-8e83-fc2d4fa66a02" path="/var/lib/kubelet/pods/a3e89b62-a248-46fc-8e83-fc2d4fa66a02/volumes"
Oct 30 13:25:08.537204 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:08.534828    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a814d069-870d-4ea0-945f-2ddcd1d517eb" path="/var/lib/kubelet/pods/a814d069-870d-4ea0-945f-2ddcd1d517eb/volumes"
Oct 30 13:25:09.906238 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:09.906200    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"]
Oct 30 13:25:09.906790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:09.906470    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerName="istio-proxy" containerID="cri-o://3e0eefdebdfca637da55f08bc21975af817b34a808d928286f5f630d7518fab0" gracePeriod=30
Oct 30 13:25:11.229832 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:11.229794    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"]
Oct 30 13:25:15.198200 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.198142    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:25:15.292838 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.292805    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-credential-socket\") pod \"68b6b218-4523-481c-8f22-b05a6e80af0a\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") "
Oct 30 13:25:15.293032 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.292876    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b6b218-4523-481c-8f22-b05a6e80af0a-kube-api-access-p47f2\" (UniqueName: \"kubernetes.io/projected/68b6b218-4523-481c-8f22-b05a6e80af0a-kube-api-access-p47f2\") pod \"68b6b218-4523-481c-8f22-b05a6e80af0a\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") "
Oct 30 13:25:15.293120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.293048    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b6b218-4523-481c-8f22-b05a6e80af0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b6b218-4523-481c-8f22-b05a6e80af0a-istiod-ca-cert\") pod \"68b6b218-4523-481c-8f22-b05a6e80af0a\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") "
Oct 30 13:25:15.293120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.293077    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-token\" (UniqueName: \"kubernetes.io/projected/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-token\") pod \"68b6b218-4523-481c-8f22-b05a6e80af0a\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") "
Oct 30 13:25:15.293120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.293109    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-workload-socket\") pod \"68b6b218-4523-481c-8f22-b05a6e80af0a\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") "
Oct 30 13:25:15.293299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.293203    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-envoy\") pod \"68b6b218-4523-481c-8f22-b05a6e80af0a\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") "
Oct 30 13:25:15.293299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.293254    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-workload-certs\") pod \"68b6b218-4523-481c-8f22-b05a6e80af0a\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") "
Oct 30 13:25:15.293414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.293396    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-podinfo\") pod \"68b6b218-4523-481c-8f22-b05a6e80af0a\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") "
Oct 30 13:25:15.294442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.293526    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-data\") pod \"68b6b218-4523-481c-8f22-b05a6e80af0a\" (UID: \"68b6b218-4523-481c-8f22-b05a6e80af0a\") "
Oct 30 13:25:15.294442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.293896    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-workload-socket" pod "68b6b218-4523-481c-8f22-b05a6e80af0a" (UID: "68b6b218-4523-481c-8f22-b05a6e80af0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:15.294442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.294177    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-workload-certs" pod "68b6b218-4523-481c-8f22-b05a6e80af0a" (UID: "68b6b218-4523-481c-8f22-b05a6e80af0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:15.294442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.294406    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-data" pod "68b6b218-4523-481c-8f22-b05a6e80af0a" (UID: "68b6b218-4523-481c-8f22-b05a6e80af0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:15.294772 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.294472    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b6b218-4523-481c-8f22-b05a6e80af0a-istiod-ca-cert" pod "68b6b218-4523-481c-8f22-b05a6e80af0a" (UID: "68b6b218-4523-481c-8f22-b05a6e80af0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:15.294910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.294784    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-credential-socket" pod "68b6b218-4523-481c-8f22-b05a6e80af0a" (UID: "68b6b218-4523-481c-8f22-b05a6e80af0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:15.295239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.295111    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.295239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.295135    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.295239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.295149    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68b6b218-4523-481c-8f22-b05a6e80af0a-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.295239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.295163    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.295239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.295175    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.298338 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.298310    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-podinfo" pod "68b6b218-4523-481c-8f22-b05a6e80af0a" (UID: "68b6b218-4523-481c-8f22-b05a6e80af0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:25:15.298726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.298672    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-envoy" pod "68b6b218-4523-481c-8f22-b05a6e80af0a" (UID: "68b6b218-4523-481c-8f22-b05a6e80af0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:15.299043 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.299022    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b6b218-4523-481c-8f22-b05a6e80af0a-kube-api-access-p47f2" pod "68b6b218-4523-481c-8f22-b05a6e80af0a" (UID: "68b6b218-4523-481c-8f22-b05a6e80af0a"). InnerVolumeSpecName "kube-api-access-p47f2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:15.299452 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.299423    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-token" pod "68b6b218-4523-481c-8f22-b05a6e80af0a" (UID: "68b6b218-4523-481c-8f22-b05a6e80af0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:15.396984 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.396953    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.396984 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.396979    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p47f2\" (UniqueName: \"kubernetes.io/projected/68b6b218-4523-481c-8f22-b05a6e80af0a-kube-api-access-p47f2\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.396984 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.396989    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.397220 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.396999    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68b6b218-4523-481c-8f22-b05a6e80af0a-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:15.560218 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.560188    5893 generic.go:356] "Generic (PLEG): container finished" podID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerID="3e0eefdebdfca637da55f08bc21975af817b34a808d928286f5f630d7518fab0" exitCode=0
Oct 30 13:25:15.560437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.560221    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" event={"ID":"68b6b218-4523-481c-8f22-b05a6e80af0a","Type":"ContainerDied","Data":"3e0eefdebdfca637da55f08bc21975af817b34a808d928286f5f630d7518fab0"}
Oct 30 13:25:15.560437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.560265    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"
Oct 30 13:25:15.560437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.560297    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v" event={"ID":"68b6b218-4523-481c-8f22-b05a6e80af0a","Type":"ContainerDied","Data":"bdfc9414b60fc579129089a22f1abcc89149653198eb663075d1ce59372df772"}
Oct 30 13:25:15.560437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.560322    5893 scope.go:122] "RemoveContainer" containerID="3e0eefdebdfca637da55f08bc21975af817b34a808d928286f5f630d7518fab0"
Oct 30 13:25:15.572330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.572248    5893 scope.go:122] "RemoveContainer" containerID="18234e525c3631c355702e50f4a6dc0e387feca51f1d515e4c4bba2bcdebc11f"
Oct 30 13:25:15.582483 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.582463    5893 scope.go:122] "RemoveContainer" containerID="3e0eefdebdfca637da55f08bc21975af817b34a808d928286f5f630d7518fab0"
Oct 30 13:25:15.582769 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:25:15.582743    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e0eefdebdfca637da55f08bc21975af817b34a808d928286f5f630d7518fab0\": container with ID starting with 3e0eefdebdfca637da55f08bc21975af817b34a808d928286f5f630d7518fab0 not found: ID does not exist" containerID="3e0eefdebdfca637da55f08bc21975af817b34a808d928286f5f630d7518fab0"
Oct 30 13:25:15.582862 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.582775    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e0eefdebdfca637da55f08bc21975af817b34a808d928286f5f630d7518fab0"} err="failed to get container status \"3e0eefdebdfca637da55f08bc21975af817b34a808d928286f5f630d7518fab0\": rpc error: code = NotFound desc = could not find container \"3e0eefdebdfca637da55f08bc21975af817b34a808d928286f5f630d7518fab0\": container with ID starting with 3e0eefdebdfca637da55f08bc21975af817b34a808d928286f5f630d7518fab0 not found: ID does not exist"
Oct 30 13:25:15.582862 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.582791    5893 scope.go:122] "RemoveContainer" containerID="18234e525c3631c355702e50f4a6dc0e387feca51f1d515e4c4bba2bcdebc11f"
Oct 30 13:25:15.583123 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:25:15.583091    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18234e525c3631c355702e50f4a6dc0e387feca51f1d515e4c4bba2bcdebc11f\": container with ID starting with 18234e525c3631c355702e50f4a6dc0e387feca51f1d515e4c4bba2bcdebc11f not found: ID does not exist" containerID="18234e525c3631c355702e50f4a6dc0e387feca51f1d515e4c4bba2bcdebc11f"
Oct 30 13:25:15.583222 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.583132    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18234e525c3631c355702e50f4a6dc0e387feca51f1d515e4c4bba2bcdebc11f"} err="failed to get container status \"18234e525c3631c355702e50f4a6dc0e387feca51f1d515e4c4bba2bcdebc11f\": rpc error: code = NotFound desc = could not find container \"18234e525c3631c355702e50f4a6dc0e387feca51f1d515e4c4bba2bcdebc11f\": container with ID starting with 18234e525c3631c355702e50f4a6dc0e387feca51f1d515e4c4bba2bcdebc11f not found: ID does not exist"
Oct 30 13:25:15.628431 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.628403    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"]
Oct 30 13:25:15.632703 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:15.632677    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-618c72-openshift-default-7d44fcfdf7-lff9v"]
Oct 30 13:25:16.399286 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:16.399232    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"]
Oct 30 13:25:16.572179 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:16.572138    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" path="/var/lib/kubelet/pods/68b6b218-4523-481c-8f22-b05a6e80af0a/volumes"
Oct 30 13:25:16.941562 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:16.941535    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"
Oct 30 13:25:17.024322 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.024288    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0db246df-9cb2-4c31-9389-13fb0c8ac395-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db246df-9cb2-4c31-9389-13fb0c8ac395-plugins-config-volume\") pod \"0db246df-9cb2-4c31-9389-13fb0c8ac395\" (UID: \"0db246df-9cb2-4c31-9389-13fb0c8ac395\") "
Oct 30 13:25:17.024527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.024395    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0db246df-9cb2-4c31-9389-13fb0c8ac395-kube-api-access-nr6wh\" (UniqueName: \"kubernetes.io/projected/0db246df-9cb2-4c31-9389-13fb0c8ac395-kube-api-access-nr6wh\") pod \"0db246df-9cb2-4c31-9389-13fb0c8ac395\" (UID: \"0db246df-9cb2-4c31-9389-13fb0c8ac395\") "
Oct 30 13:25:17.024714 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.024687    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0db246df-9cb2-4c31-9389-13fb0c8ac395-plugins-config-volume" pod "0db246df-9cb2-4c31-9389-13fb0c8ac395" (UID: "0db246df-9cb2-4c31-9389-13fb0c8ac395"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:17.025325 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.025296    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0db246df-9cb2-4c31-9389-13fb0c8ac395-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:17.027202 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.027176    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0db246df-9cb2-4c31-9389-13fb0c8ac395-kube-api-access-nr6wh" pod "0db246df-9cb2-4c31-9389-13fb0c8ac395" (UID: "0db246df-9cb2-4c31-9389-13fb0c8ac395"). InnerVolumeSpecName "kube-api-access-nr6wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:17.126239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.126202    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr6wh\" (UniqueName: \"kubernetes.io/projected/0db246df-9cb2-4c31-9389-13fb0c8ac395-kube-api-access-nr6wh\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:17.589249 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.589211    5893 generic.go:356] "Generic (PLEG): container finished" podID="0db246df-9cb2-4c31-9389-13fb0c8ac395" containerID="16747b4a97b604f70df517f77ff7bb6f9d51ab478fa89e49d34cd29cf6b4da1a" exitCode=0
Oct 30 13:25:17.589790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.589265    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"
Oct 30 13:25:17.589790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.589303    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr" event={"ID":"0db246df-9cb2-4c31-9389-13fb0c8ac395","Type":"ContainerDied","Data":"16747b4a97b604f70df517f77ff7bb6f9d51ab478fa89e49d34cd29cf6b4da1a"}
Oct 30 13:25:17.589790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.589363    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr" event={"ID":"0db246df-9cb2-4c31-9389-13fb0c8ac395","Type":"ContainerDied","Data":"57ace622c0f98727b1f270036c43462b642f90fa9a48cb5220d55393f313bed4"}
Oct 30 13:25:17.589790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.589384    5893 scope.go:122] "RemoveContainer" containerID="16747b4a97b604f70df517f77ff7bb6f9d51ab478fa89e49d34cd29cf6b4da1a"
Oct 30 13:25:17.600727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.600706    5893 scope.go:122] "RemoveContainer" containerID="16747b4a97b604f70df517f77ff7bb6f9d51ab478fa89e49d34cd29cf6b4da1a"
Oct 30 13:25:17.600998 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:25:17.600979    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16747b4a97b604f70df517f77ff7bb6f9d51ab478fa89e49d34cd29cf6b4da1a\": container with ID starting with 16747b4a97b604f70df517f77ff7bb6f9d51ab478fa89e49d34cd29cf6b4da1a not found: ID does not exist" containerID="16747b4a97b604f70df517f77ff7bb6f9d51ab478fa89e49d34cd29cf6b4da1a"
Oct 30 13:25:17.601079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.601006    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16747b4a97b604f70df517f77ff7bb6f9d51ab478fa89e49d34cd29cf6b4da1a"} err="failed to get container status \"16747b4a97b604f70df517f77ff7bb6f9d51ab478fa89e49d34cd29cf6b4da1a\": rpc error: code = NotFound desc = could not find container \"16747b4a97b604f70df517f77ff7bb6f9d51ab478fa89e49d34cd29cf6b4da1a\": container with ID starting with 16747b4a97b604f70df517f77ff7bb6f9d51ab478fa89e49d34cd29cf6b4da1a not found: ID does not exist"
Oct 30 13:25:17.649351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.649312    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"]
Oct 30 13:25:17.659222 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:17.659193    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bec983-66679fd98c-4mflr"]
Oct 30 13:25:18.533245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:18.533212    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0db246df-9cb2-4c31-9389-13fb0c8ac395" path="/var/lib/kubelet/pods/0db246df-9cb2-4c31-9389-13fb0c8ac395/volumes"
Oct 30 13:25:18.646485 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:25:18.646448    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:25:20.940223 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:20.940191    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"]
Oct 30 13:25:20.991017 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:20.988003    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"]
Oct 30 13:25:25.941289 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:25.941223    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk" podUID="41ef6587-02b7-4fa4-9225-d090977e2310" containerName="payload-processing" containerID="cri-o://519200e0bece3bacdd11a2bc5cd82215171b2d1a93331df2b5913c4b4558f750" gracePeriod=25
Oct 30 13:25:25.989679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:25.989583    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2" podUID="bfe59aa0-d066-43f1-97de-9fa4775c71bf" containerName="payload-pre-processing" containerID="cri-o://c18fd38beac63be783eca55f565a520b3f5e78bede233cc24406f2f0c2626f98" gracePeriod=25
Oct 30 13:25:26.242102 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.242075    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"
Oct 30 13:25:26.252045 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.252008    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw" podUID="58298a7c-0dd3-474b-a5ae-263fae7c9e51" containerName="main" containerID="cri-o://780db5298856c44ae984b3252efc810efeb97cac71e8576eb6efcbebd2f8012b" gracePeriod=45
Oct 30 13:25:26.294685 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.294644    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ef6587-02b7-4fa4-9225-d090977e2310-kube-api-access-2xq24\" (UniqueName: \"kubernetes.io/projected/41ef6587-02b7-4fa4-9225-d090977e2310-kube-api-access-2xq24\") pod \"41ef6587-02b7-4fa4-9225-d090977e2310\" (UID: \"41ef6587-02b7-4fa4-9225-d090977e2310\") "
Oct 30 13:25:26.294857 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.294752    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41ef6587-02b7-4fa4-9225-d090977e2310-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41ef6587-02b7-4fa4-9225-d090977e2310-plugins-config-volume\") pod \"41ef6587-02b7-4fa4-9225-d090977e2310\" (UID: \"41ef6587-02b7-4fa4-9225-d090977e2310\") "
Oct 30 13:25:26.295433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.295403    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ef6587-02b7-4fa4-9225-d090977e2310-plugins-config-volume" pod "41ef6587-02b7-4fa4-9225-d090977e2310" (UID: "41ef6587-02b7-4fa4-9225-d090977e2310"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:26.297156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.297127    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ef6587-02b7-4fa4-9225-d090977e2310-kube-api-access-2xq24" pod "41ef6587-02b7-4fa4-9225-d090977e2310" (UID: "41ef6587-02b7-4fa4-9225-d090977e2310"). InnerVolumeSpecName "kube-api-access-2xq24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:26.396790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.396753    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xq24\" (UniqueName: \"kubernetes.io/projected/41ef6587-02b7-4fa4-9225-d090977e2310-kube-api-access-2xq24\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:26.396790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.396786    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41ef6587-02b7-4fa4-9225-d090977e2310-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:26.493831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.493795    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"]
Oct 30 13:25:26.494102 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.494073    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc" podUID="40116e5b-e2b3-45b8-93e9-a0faa7fbe950" containerName="maas-api" containerID="cri-o://4a647f6fbffa7fd77823e0e7b4068ae4d6fb058dfc15808c0b327bfca0ae5431" gracePeriod=30
Oct 30 13:25:26.610072 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.610041    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:25:26.671608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.670008    5893 generic.go:356] "Generic (PLEG): container finished" podID="41ef6587-02b7-4fa4-9225-d090977e2310" containerID="519200e0bece3bacdd11a2bc5cd82215171b2d1a93331df2b5913c4b4558f750" exitCode=0
Oct 30 13:25:26.671608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.670121    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk" event={"ID":"41ef6587-02b7-4fa4-9225-d090977e2310","Type":"ContainerDied","Data":"519200e0bece3bacdd11a2bc5cd82215171b2d1a93331df2b5913c4b4558f750"}
Oct 30 13:25:26.671608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.670172    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk" event={"ID":"41ef6587-02b7-4fa4-9225-d090977e2310","Type":"ContainerDied","Data":"29475e5efa82c287083eb93292b5b62cfbd3264e327ddc197828f37510f2d42d"}
Oct 30 13:25:26.671608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.670205    5893 scope.go:122] "RemoveContainer" containerID="519200e0bece3bacdd11a2bc5cd82215171b2d1a93331df2b5913c4b4558f750"
Oct 30 13:25:26.671608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.670368    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"
Oct 30 13:25:26.677365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.676944    5893 generic.go:356] "Generic (PLEG): container finished" podID="58298a7c-0dd3-474b-a5ae-263fae7c9e51" containerID="780db5298856c44ae984b3252efc810efeb97cac71e8576eb6efcbebd2f8012b" exitCode=0
Oct 30 13:25:26.677365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.677078    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw" event={"ID":"58298a7c-0dd3-474b-a5ae-263fae7c9e51","Type":"ContainerDied","Data":"780db5298856c44ae984b3252efc810efeb97cac71e8576eb6efcbebd2f8012b"}
Oct 30 13:25:26.677365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.677116    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw" event={"ID":"58298a7c-0dd3-474b-a5ae-263fae7c9e51","Type":"ContainerDied","Data":"406a5a312d93463862e5271298677ad94d678cf5fd75a0fcba355b0bf997be69"}
Oct 30 13:25:26.677365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.677194    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"
Oct 30 13:25:26.679704 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.679562    5893 generic.go:356] "Generic (PLEG): container finished" podID="40116e5b-e2b3-45b8-93e9-a0faa7fbe950" containerID="4a647f6fbffa7fd77823e0e7b4068ae4d6fb058dfc15808c0b327bfca0ae5431" exitCode=0
Oct 30 13:25:26.679704 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.679651    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc" event={"ID":"40116e5b-e2b3-45b8-93e9-a0faa7fbe950","Type":"ContainerDied","Data":"4a647f6fbffa7fd77823e0e7b4068ae4d6fb058dfc15808c0b327bfca0ae5431"}
Oct 30 13:25:26.692917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.692229    5893 scope.go:122] "RemoveContainer" containerID="519200e0bece3bacdd11a2bc5cd82215171b2d1a93331df2b5913c4b4558f750"
Oct 30 13:25:26.692917 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:25:26.692578    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519200e0bece3bacdd11a2bc5cd82215171b2d1a93331df2b5913c4b4558f750\": container with ID starting with 519200e0bece3bacdd11a2bc5cd82215171b2d1a93331df2b5913c4b4558f750 not found: ID does not exist" containerID="519200e0bece3bacdd11a2bc5cd82215171b2d1a93331df2b5913c4b4558f750"
Oct 30 13:25:26.692917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.692606    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519200e0bece3bacdd11a2bc5cd82215171b2d1a93331df2b5913c4b4558f750"} err="failed to get container status \"519200e0bece3bacdd11a2bc5cd82215171b2d1a93331df2b5913c4b4558f750\": rpc error: code = NotFound desc = could not find container \"519200e0bece3bacdd11a2bc5cd82215171b2d1a93331df2b5913c4b4558f750\": container with ID starting with 519200e0bece3bacdd11a2bc5cd82215171b2d1a93331df2b5913c4b4558f750 not found: ID does not exist"
Oct 30 13:25:26.692917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.692622    5893 scope.go:122] "RemoveContainer" containerID="780db5298856c44ae984b3252efc810efeb97cac71e8576eb6efcbebd2f8012b"
Oct 30 13:25:26.710964 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.710942    5893 scope.go:122] "RemoveContainer" containerID="780db5298856c44ae984b3252efc810efeb97cac71e8576eb6efcbebd2f8012b"
Oct 30 13:25:26.711314 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:25:26.711285    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780db5298856c44ae984b3252efc810efeb97cac71e8576eb6efcbebd2f8012b\": container with ID starting with 780db5298856c44ae984b3252efc810efeb97cac71e8576eb6efcbebd2f8012b not found: ID does not exist" containerID="780db5298856c44ae984b3252efc810efeb97cac71e8576eb6efcbebd2f8012b"
Oct 30 13:25:26.711408 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.711325    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780db5298856c44ae984b3252efc810efeb97cac71e8576eb6efcbebd2f8012b"} err="failed to get container status \"780db5298856c44ae984b3252efc810efeb97cac71e8576eb6efcbebd2f8012b\": rpc error: code = NotFound desc = could not find container \"780db5298856c44ae984b3252efc810efeb97cac71e8576eb6efcbebd2f8012b\": container with ID starting with 780db5298856c44ae984b3252efc810efeb97cac71e8576eb6efcbebd2f8012b not found: ID does not exist"
Oct 30 13:25:26.724238 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.724212    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-home\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-home\") pod \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") "
Oct 30 13:25:26.724401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.724259    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58298a7c-0dd3-474b-a5ae-263fae7c9e51-tls-certs\" (UniqueName: \"kubernetes.io/secret/58298a7c-0dd3-474b-a5ae-263fae7c9e51-tls-certs\") pod \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") "
Oct 30 13:25:26.724401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.724345    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-tmp-dir\") pod \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") "
Oct 30 13:25:26.724401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.724377    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-dshm\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-dshm\") pod \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") "
Oct 30 13:25:26.724550 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.724523    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58298a7c-0dd3-474b-a5ae-263fae7c9e51-kube-api-access-wz2fq\" (UniqueName: \"kubernetes.io/projected/58298a7c-0dd3-474b-a5ae-263fae7c9e51-kube-api-access-wz2fq\") pod \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") "
Oct 30 13:25:26.724609 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.724594    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-model-cache\") pod \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\" (UID: \"58298a7c-0dd3-474b-a5ae-263fae7c9e51\") "
Oct 30 13:25:26.727890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.725904    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-model-cache" pod "58298a7c-0dd3-474b-a5ae-263fae7c9e51" (UID: "58298a7c-0dd3-474b-a5ae-263fae7c9e51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:26.728795 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.728756    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-tmp-dir" pod "58298a7c-0dd3-474b-a5ae-263fae7c9e51" (UID: "58298a7c-0dd3-474b-a5ae-263fae7c9e51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:26.729032 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.729005    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-home" pod "58298a7c-0dd3-474b-a5ae-263fae7c9e51" (UID: "58298a7c-0dd3-474b-a5ae-263fae7c9e51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:26.755710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.751378    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"]
Oct 30 13:25:26.776309 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.774499    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-619b9b-5585886b4d-rf2wk"]
Oct 30 13:25:26.834359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.816059    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58298a7c-0dd3-474b-a5ae-263fae7c9e51-tls-certs" pod "58298a7c-0dd3-474b-a5ae-263fae7c9e51" (UID: "58298a7c-0dd3-474b-a5ae-263fae7c9e51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:26.834359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.816240    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58298a7c-0dd3-474b-a5ae-263fae7c9e51-kube-api-access-wz2fq" pod "58298a7c-0dd3-474b-a5ae-263fae7c9e51" (UID: "58298a7c-0dd3-474b-a5ae-263fae7c9e51"). InnerVolumeSpecName "kube-api-access-wz2fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:26.834359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.816645    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-dshm" pod "58298a7c-0dd3-474b-a5ae-263fae7c9e51" (UID: "58298a7c-0dd3-474b-a5ae-263fae7c9e51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:26.834359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.829497    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:26.834359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.829532    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58298a7c-0dd3-474b-a5ae-263fae7c9e51-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:26.834359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.829549    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:26.834359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.829561    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:26.834359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.829576    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz2fq\" (UniqueName: \"kubernetes.io/projected/58298a7c-0dd3-474b-a5ae-263fae7c9e51-kube-api-access-wz2fq\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:26.834359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.829590    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58298a7c-0dd3-474b-a5ae-263fae7c9e51-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:26.861884 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.861856    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:25:26.939800 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.939762    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-maas-api-tls\") pod \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") "
Oct 30 13:25:26.940018 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.939920    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-kube-api-access-pbnkg\" (UniqueName: \"kubernetes.io/projected/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-kube-api-access-pbnkg\") pod \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") "
Oct 30 13:25:26.940018 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.939971    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-metrics-certs\") pod \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") "
Oct 30 13:25:26.940217 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.940032    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-openshift-service-ca\") pod \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\" (UID: \"40116e5b-e2b3-45b8-93e9-a0faa7fbe950\") "
Oct 30 13:25:26.966684 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:26.965508    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-openshift-service-ca" pod "40116e5b-e2b3-45b8-93e9-a0faa7fbe950" (UID: "40116e5b-e2b3-45b8-93e9-a0faa7fbe950"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:27.060186 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.045472    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:27.064788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.064726    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-kube-api-access-pbnkg" pod "40116e5b-e2b3-45b8-93e9-a0faa7fbe950" (UID: "40116e5b-e2b3-45b8-93e9-a0faa7fbe950"). InnerVolumeSpecName "kube-api-access-pbnkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:27.064788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.064767    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-maas-api-tls" pod "40116e5b-e2b3-45b8-93e9-a0faa7fbe950" (UID: "40116e5b-e2b3-45b8-93e9-a0faa7fbe950"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:27.067395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.067257    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-metrics-certs" pod "40116e5b-e2b3-45b8-93e9-a0faa7fbe950" (UID: "40116e5b-e2b3-45b8-93e9-a0faa7fbe950"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:27.123374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.123338    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"]
Oct 30 13:25:27.126186 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.126165    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw"]
Oct 30 13:25:27.146969 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.146933    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:27.146969 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.146966    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbnkg\" (UniqueName: \"kubernetes.io/projected/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-kube-api-access-pbnkg\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:27.147163 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.146981    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40116e5b-e2b3-45b8-93e9-a0faa7fbe950-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:27.488818 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.488770    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-acd125/auth-test-model-acd125-kserve-5c5969b4fc-p6glw" podUID="58298a7c-0dd3-474b-a5ae-263fae7c9e51" containerName="main" probeResult="failure" output="Get \"https://10.134.0.142:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:25:27.695048 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.695004    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc" event={"ID":"40116e5b-e2b3-45b8-93e9-a0faa7fbe950","Type":"ContainerDied","Data":"c5ddeae0b68c6498feb24b5afe94d200aeece109b15dc39bdd85f72e383931fb"}
Oct 30 13:25:27.695234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.695072    5893 scope.go:122] "RemoveContainer" containerID="4a647f6fbffa7fd77823e0e7b4068ae4d6fb058dfc15808c0b327bfca0ae5431"
Oct 30 13:25:27.695360 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.695341    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"
Oct 30 13:25:27.751046 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.750964    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"]
Oct 30 13:25:27.760350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:27.755686    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-bec983-549c44dd6-fcsqc"]
Oct 30 13:25:28.443235 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:28.443197    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2" podUID="bfe59aa0-d066-43f1-97de-9fa4775c71bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:25:28.534793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:28.534757    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40116e5b-e2b3-45b8-93e9-a0faa7fbe950" path="/var/lib/kubelet/pods/40116e5b-e2b3-45b8-93e9-a0faa7fbe950/volumes"
Oct 30 13:25:28.535371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:28.535349    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ef6587-02b7-4fa4-9225-d090977e2310" path="/var/lib/kubelet/pods/41ef6587-02b7-4fa4-9225-d090977e2310/volumes"
Oct 30 13:25:28.535939 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:28.535919    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58298a7c-0dd3-474b-a5ae-263fae7c9e51" path="/var/lib/kubelet/pods/58298a7c-0dd3-474b-a5ae-263fae7c9e51/volumes"
Oct 30 13:25:31.423523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.423482    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg" podUID="52ba1504-83f4-4592-9b3e-e5be4ba8e20a" containerName="main" containerID="cri-o://6077defc75db97b0d3cf0a60e42aa665d8c8f3719520e04f79fce6669eb4fa22" gracePeriod=45
Oct 30 13:25:31.826720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.826378    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:25:31.828922 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.828891    5893 generic.go:356] "Generic (PLEG): container finished" podID="52ba1504-83f4-4592-9b3e-e5be4ba8e20a" containerID="6077defc75db97b0d3cf0a60e42aa665d8c8f3719520e04f79fce6669eb4fa22" exitCode=0
Oct 30 13:25:31.829261 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.829242    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg" event={"ID":"52ba1504-83f4-4592-9b3e-e5be4ba8e20a","Type":"ContainerDied","Data":"6077defc75db97b0d3cf0a60e42aa665d8c8f3719520e04f79fce6669eb4fa22"}
Oct 30 13:25:31.829413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.829398    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg" event={"ID":"52ba1504-83f4-4592-9b3e-e5be4ba8e20a","Type":"ContainerDied","Data":"9376bc9b994dffe7e7927c7be30186e433e003650f8e1177f42384e2e63faa99"}
Oct 30 13:25:31.829536 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.829524    5893 scope.go:122] "RemoveContainer" containerID="6077defc75db97b0d3cf0a60e42aa665d8c8f3719520e04f79fce6669eb4fa22"
Oct 30 13:25:31.860417 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.860073    5893 scope.go:122] "RemoveContainer" containerID="6077defc75db97b0d3cf0a60e42aa665d8c8f3719520e04f79fce6669eb4fa22"
Oct 30 13:25:31.860779 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:25:31.860675    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6077defc75db97b0d3cf0a60e42aa665d8c8f3719520e04f79fce6669eb4fa22\": container with ID starting with 6077defc75db97b0d3cf0a60e42aa665d8c8f3719520e04f79fce6669eb4fa22 not found: ID does not exist" containerID="6077defc75db97b0d3cf0a60e42aa665d8c8f3719520e04f79fce6669eb4fa22"
Oct 30 13:25:31.860779 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.860715    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6077defc75db97b0d3cf0a60e42aa665d8c8f3719520e04f79fce6669eb4fa22"} err="failed to get container status \"6077defc75db97b0d3cf0a60e42aa665d8c8f3719520e04f79fce6669eb4fa22\": rpc error: code = NotFound desc = could not find container \"6077defc75db97b0d3cf0a60e42aa665d8c8f3719520e04f79fce6669eb4fa22\": container with ID starting with 6077defc75db97b0d3cf0a60e42aa665d8c8f3719520e04f79fce6669eb4fa22 not found: ID does not exist"
Oct 30 13:25:31.937960 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.937850    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-kube-api-access-6gm2j\" (UniqueName: \"kubernetes.io/projected/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-kube-api-access-6gm2j\") pod \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") "
Oct 30 13:25:31.937960 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.937947    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-model-cache\") pod \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") "
Oct 30 13:25:31.938236 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.938007    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-dshm\") pod \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") "
Oct 30 13:25:31.938236 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.938124    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-home\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-home\") pod \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") "
Oct 30 13:25:31.938236 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.938182    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-tmp-dir\") pod \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") "
Oct 30 13:25:31.938438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.938345    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-tls-certs\" (UniqueName: \"kubernetes.io/secret/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-tls-certs\") pod \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\" (UID: \"52ba1504-83f4-4592-9b3e-e5be4ba8e20a\") "
Oct 30 13:25:31.939791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.939240    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-home" pod "52ba1504-83f4-4592-9b3e-e5be4ba8e20a" (UID: "52ba1504-83f4-4592-9b3e-e5be4ba8e20a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:31.939791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.939452    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-model-cache" pod "52ba1504-83f4-4592-9b3e-e5be4ba8e20a" (UID: "52ba1504-83f4-4592-9b3e-e5be4ba8e20a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:31.939791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.939536    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:31.939791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.939677    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-tmp-dir" pod "52ba1504-83f4-4592-9b3e-e5be4ba8e20a" (UID: "52ba1504-83f4-4592-9b3e-e5be4ba8e20a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:31.943011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.942540    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-tls-certs" pod "52ba1504-83f4-4592-9b3e-e5be4ba8e20a" (UID: "52ba1504-83f4-4592-9b3e-e5be4ba8e20a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:31.943011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.942572    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-dshm" pod "52ba1504-83f4-4592-9b3e-e5be4ba8e20a" (UID: "52ba1504-83f4-4592-9b3e-e5be4ba8e20a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:31.943011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:31.942956    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-kube-api-access-6gm2j" pod "52ba1504-83f4-4592-9b3e-e5be4ba8e20a" (UID: "52ba1504-83f4-4592-9b3e-e5be4ba8e20a"). InnerVolumeSpecName "kube-api-access-6gm2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:32.041343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:32.041226    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:32.041343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:32.041255    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gm2j\" (UniqueName: \"kubernetes.io/projected/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-kube-api-access-6gm2j\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:32.041343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:32.041265    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:32.041343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:32.041293    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:32.041343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:32.041304    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52ba1504-83f4-4592-9b3e-e5be4ba8e20a-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:32.837523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:32.837437    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"
Oct 30 13:25:32.897164 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:32.897067    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"]
Oct 30 13:25:32.909025 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:32.908984    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-e9d641/auth-test-model-e9d641-kserve-855bb7d4f-p9sxg"]
Oct 30 13:25:34.534544 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:34.534513    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ba1504-83f4-4592-9b3e-e5be4ba8e20a" path="/var/lib/kubelet/pods/52ba1504-83f4-4592-9b3e-e5be4ba8e20a/volumes"
Oct 30 13:25:34.688863 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:34.688830    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"]
Oct 30 13:25:34.689049 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:34.689025    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv" podUID="09c179d0-e114-4d43-b296-db22061f87d6" containerName="istio-proxy" containerID="cri-o://78e8e3b484080071adffbe3ecb1874c595ba6887626ac219ea2c7be910543478" gracePeriod=30
Oct 30 13:25:35.002590 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.002558    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9077-ab59dd"]
Oct 30 13:25:35.005093 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.005068    5893 cpu_manager.go:414] "containerMap: removing container" podUID="58298a7c-0dd3-474b-a5ae-263fae7c9e51" containerName="main"
Oct 30 13:25:35.005240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.005228    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58298a7c-0dd3-474b-a5ae-263fae7c9e51" containerName="main"
Oct 30 13:25:35.005421 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.005409    5893 cpu_manager.go:414] "containerMap: removing container" podUID="967ac404-7b3d-49bd-9cb5-63e148bb4f9a" containerName="payload-processing"
Oct 30 13:25:35.005512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.005502    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967ac404-7b3d-49bd-9cb5-63e148bb4f9a" containerName="payload-processing"
Oct 30 13:25:35.005647 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.005637    5893 cpu_manager.go:414] "containerMap: removing container" podUID="52ba1504-83f4-4592-9b3e-e5be4ba8e20a" containerName="main"
Oct 30 13:25:35.005725 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.005716    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ba1504-83f4-4592-9b3e-e5be4ba8e20a" containerName="main"
Oct 30 13:25:35.005809 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.005801    5893 cpu_manager.go:414] "containerMap: removing container" podUID="40116e5b-e2b3-45b8-93e9-a0faa7fbe950" containerName="maas-api"
Oct 30 13:25:35.005874 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.005865    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40116e5b-e2b3-45b8-93e9-a0faa7fbe950" containerName="maas-api"
Oct 30 13:25:35.005993 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.005983    5893 cpu_manager.go:414] "containerMap: removing container" podUID="41ef6587-02b7-4fa4-9225-d090977e2310" containerName="payload-processing"
Oct 30 13:25:35.006061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.006052    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41ef6587-02b7-4fa4-9225-d090977e2310" containerName="payload-processing"
Oct 30 13:25:35.006149 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.006140    5893 cpu_manager.go:414] "containerMap: removing container" podUID="0db246df-9cb2-4c31-9389-13fb0c8ac395" containerName="payload-pre-processing"
Oct 30 13:25:35.006214 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.006204    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0db246df-9cb2-4c31-9389-13fb0c8ac395" containerName="payload-pre-processing"
Oct 30 13:25:35.006310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.006300    5893 cpu_manager.go:414] "containerMap: removing container" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerName="istio-proxy"
Oct 30 13:25:35.006378 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.006370    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerName="istio-proxy"
Oct 30 13:25:35.006494 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.006485    5893 cpu_manager.go:414] "containerMap: removing container" podUID="a3e89b62-a248-46fc-8e83-fc2d4fa66a02" containerName="payload-processing"
Oct 30 13:25:35.006556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.006548    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3e89b62-a248-46fc-8e83-fc2d4fa66a02" containerName="payload-processing"
Oct 30 13:25:35.006645 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.006636    5893 cpu_manager.go:414] "containerMap: removing container" podUID="379ef92e-3861-4eec-8eb2-c52787142d1a" containerName="istio-proxy"
Oct 30 13:25:35.006706 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.006698    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="379ef92e-3861-4eec-8eb2-c52787142d1a" containerName="istio-proxy"
Oct 30 13:25:35.006777 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.006769    5893 cpu_manager.go:414] "containerMap: removing container" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerName="istio-proxy"
Oct 30 13:25:35.006834 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.006826    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerName="istio-proxy"
Oct 30 13:25:35.006892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.006885    5893 cpu_manager.go:414] "containerMap: removing container" podUID="a814d069-870d-4ea0-945f-2ddcd1d517eb" containerName="payload-pre-processing"
Oct 30 13:25:35.006956 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.006947    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a814d069-870d-4ea0-945f-2ddcd1d517eb" containerName="payload-pre-processing"
Oct 30 13:25:35.007345 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.007327    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="967ac404-7b3d-49bd-9cb5-63e148bb4f9a" containerName="payload-processing"
Oct 30 13:25:35.007482 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.007471    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ba1504-83f4-4592-9b3e-e5be4ba8e20a" containerName="main"
Oct 30 13:25:35.007562 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.007554    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="40116e5b-e2b3-45b8-93e9-a0faa7fbe950" containerName="maas-api"
Oct 30 13:25:35.007722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.007711    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="41ef6587-02b7-4fa4-9225-d090977e2310" containerName="payload-processing"
Oct 30 13:25:35.007805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.007797    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="0db246df-9cb2-4c31-9389-13fb0c8ac395" containerName="payload-pre-processing"
Oct 30 13:25:35.007889 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.007880    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerName="istio-proxy"
Oct 30 13:25:35.007972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.007963    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3e89b62-a248-46fc-8e83-fc2d4fa66a02" containerName="payload-processing"
Oct 30 13:25:35.008037 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.008030    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="379ef92e-3861-4eec-8eb2-c52787142d1a" containerName="istio-proxy"
Oct 30 13:25:35.008114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.008106    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b6b218-4523-481c-8f22-b05a6e80af0a" containerName="istio-proxy"
Oct 30 13:25:35.008185 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.008178    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="a814d069-870d-4ea0-945f-2ddcd1d517eb" containerName="payload-pre-processing"
Oct 30 13:25:35.008256 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.008248    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="58298a7c-0dd3-474b-a5ae-263fae7c9e51" containerName="main"
Oct 30 13:25:35.014734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.014715    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9077-ab59dd"]
Oct 30 13:25:35.014963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.014947    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-ab59dd"
Oct 30 13:25:35.031557 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.031534    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8xhdg\""
Oct 30 13:25:35.097360 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.097321    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcwtq\" (UniqueName: \"kubernetes.io/projected/ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8-kube-api-access-qcwtq\") pod \"test-curl-9077-ab59dd\" (UID: \"ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8\") " pod="opendatahub/test-curl-9077-ab59dd"
Oct 30 13:25:35.199481 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.199449    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcwtq\" (UniqueName: \"kubernetes.io/projected/ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8-kube-api-access-qcwtq\") pod \"test-curl-9077-ab59dd\" (UID: \"ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8\") " pod="opendatahub/test-curl-9077-ab59dd"
Oct 30 13:25:35.222467 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.222439    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcwtq\" (UniqueName: \"kubernetes.io/projected/ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8-kube-api-access-qcwtq\") pod \"test-curl-9077-ab59dd\" (UID: \"ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8\") " pod="opendatahub/test-curl-9077-ab59dd"
Oct 30 13:25:35.333775 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.333693    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-ab59dd"
Oct 30 13:25:35.678730 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.678698    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9077-ab59dd"]
Oct 30 13:25:35.679136 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:25:35.679022    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded69cbe6_afe1_46a3_be2f_9c6d40dd5ff8.slice/crio-5e49e6d7ee5adb414795bcf76c38dba30f6cf0ee2ae7d75aaf2badbc32608519 WatchSource:0}: Error finding container 5e49e6d7ee5adb414795bcf76c38dba30f6cf0ee2ae7d75aaf2badbc32608519: Status 404 returned error can't find the container with id 5e49e6d7ee5adb414795bcf76c38dba30f6cf0ee2ae7d75aaf2badbc32608519
Oct 30 13:25:35.681608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:35.681587    5893 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:25:36.011401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:36.011362    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9077-ab59dd" event={"ID":"ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8","Type":"ContainerStarted","Data":"5e49e6d7ee5adb414795bcf76c38dba30f6cf0ee2ae7d75aaf2badbc32608519"}
Oct 30 13:25:37.019741 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:37.019707    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9077-ab59dd" event={"ID":"ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8","Type":"ContainerStarted","Data":"680f465ec27f2cc753a592ae5eae934cd806a11ddf6378d5a631e885aa656676"}
Oct 30 13:25:38.079897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:38.079824    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9077-ab59dd" podStartSLOduration=3.453945927 podStartE2EDuration="4.07980509s" totalImagesPullingTime="625.859163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:37.070472753 +0000 UTC m=+2378.653902448" watchObservedRunningTime="2026-10-30 13:25:38.07980509 +0000 UTC m=+2379.663234785"
Oct 30 13:25:38.083094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:38.083065    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9077-ab59dd"]
Oct 30 13:25:38.088888 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:38.088860    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9077-ab59dd"]
Oct 30 13:25:38.443898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:38.443806    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2" podUID="bfe59aa0-d066-43f1-97de-9fa4775c71bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:25:39.034351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:39.034309    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9077-ab59dd" podUID="ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8" containerName="test-curl-9077-ab59dd" containerID="cri-o://680f465ec27f2cc753a592ae5eae934cd806a11ddf6378d5a631e885aa656676" gracePeriod=2
Oct 30 13:25:40.009859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.009823    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:25:40.044943 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.044908    5893 generic.go:356] "Generic (PLEG): container finished" podID="09c179d0-e114-4d43-b296-db22061f87d6" containerID="78e8e3b484080071adffbe3ecb1874c595ba6887626ac219ea2c7be910543478" exitCode=0
Oct 30 13:25:40.045141 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.045028    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv" event={"ID":"09c179d0-e114-4d43-b296-db22061f87d6","Type":"ContainerDied","Data":"78e8e3b484080071adffbe3ecb1874c595ba6887626ac219ea2c7be910543478"}
Oct 30 13:25:40.045141 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.045101    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv" event={"ID":"09c179d0-e114-4d43-b296-db22061f87d6","Type":"ContainerDied","Data":"eddd21726c2ec82efb4f76b539fe98750729ed4c36e302e35fd5391506f7c966"}
Oct 30 13:25:40.045141 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.045121    5893 scope.go:122] "RemoveContainer" containerID="78e8e3b484080071adffbe3ecb1874c595ba6887626ac219ea2c7be910543478"
Oct 30 13:25:40.045338 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.045258    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"
Oct 30 13:25:40.063798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.063686    5893 scope.go:122] "RemoveContainer" containerID="78e8e3b484080071adffbe3ecb1874c595ba6887626ac219ea2c7be910543478"
Oct 30 13:25:40.076331 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:25:40.075832    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78e8e3b484080071adffbe3ecb1874c595ba6887626ac219ea2c7be910543478\": container with ID starting with 78e8e3b484080071adffbe3ecb1874c595ba6887626ac219ea2c7be910543478 not found: ID does not exist" containerID="78e8e3b484080071adffbe3ecb1874c595ba6887626ac219ea2c7be910543478"
Oct 30 13:25:40.076331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.075917    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78e8e3b484080071adffbe3ecb1874c595ba6887626ac219ea2c7be910543478"} err="failed to get container status \"78e8e3b484080071adffbe3ecb1874c595ba6887626ac219ea2c7be910543478\": rpc error: code = NotFound desc = could not find container \"78e8e3b484080071adffbe3ecb1874c595ba6887626ac219ea2c7be910543478\": container with ID starting with 78e8e3b484080071adffbe3ecb1874c595ba6887626ac219ea2c7be910543478 not found: ID does not exist"
Oct 30 13:25:40.124131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.123095    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-workload-certs\") pod \"09c179d0-e114-4d43-b296-db22061f87d6\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") "
Oct 30 13:25:40.124131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.123153    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-credential-socket\") pod \"09c179d0-e114-4d43-b296-db22061f87d6\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") "
Oct 30 13:25:40.124131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.123218    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-istio-data\") pod \"09c179d0-e114-4d43-b296-db22061f87d6\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") "
Oct 30 13:25:40.124131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.123391    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-istio-envoy\") pod \"09c179d0-e114-4d43-b296-db22061f87d6\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") "
Oct 30 13:25:40.124131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.123473    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c179d0-e114-4d43-b296-db22061f87d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c179d0-e114-4d43-b296-db22061f87d6-istiod-ca-cert\") pod \"09c179d0-e114-4d43-b296-db22061f87d6\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") "
Oct 30 13:25:40.124131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.123560    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c179d0-e114-4d43-b296-db22061f87d6-kube-api-access-28z8w\" (UniqueName: \"kubernetes.io/projected/09c179d0-e114-4d43-b296-db22061f87d6-kube-api-access-28z8w\") pod \"09c179d0-e114-4d43-b296-db22061f87d6\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") "
Oct 30 13:25:40.124131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.123612    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c179d0-e114-4d43-b296-db22061f87d6-istio-token\" (UniqueName: \"kubernetes.io/projected/09c179d0-e114-4d43-b296-db22061f87d6-istio-token\") pod \"09c179d0-e114-4d43-b296-db22061f87d6\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") "
Oct 30 13:25:40.124131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.123659    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-workload-socket\") pod \"09c179d0-e114-4d43-b296-db22061f87d6\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") "
Oct 30 13:25:40.124131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.123788    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09c179d0-e114-4d43-b296-db22061f87d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c179d0-e114-4d43-b296-db22061f87d6-istio-podinfo\") pod \"09c179d0-e114-4d43-b296-db22061f87d6\" (UID: \"09c179d0-e114-4d43-b296-db22061f87d6\") "
Oct 30 13:25:40.126880 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.126783    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-istio-data" pod "09c179d0-e114-4d43-b296-db22061f87d6" (UID: "09c179d0-e114-4d43-b296-db22061f87d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:40.127066 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.127038    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-workload-socket" pod "09c179d0-e114-4d43-b296-db22061f87d6" (UID: "09c179d0-e114-4d43-b296-db22061f87d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:40.127229 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.127206    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-workload-certs" pod "09c179d0-e114-4d43-b296-db22061f87d6" (UID: "09c179d0-e114-4d43-b296-db22061f87d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:40.127581 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.127384    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c179d0-e114-4d43-b296-db22061f87d6-istiod-ca-cert" pod "09c179d0-e114-4d43-b296-db22061f87d6" (UID: "09c179d0-e114-4d43-b296-db22061f87d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:40.128808 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.128770    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09c179d0-e114-4d43-b296-db22061f87d6-istio-podinfo" pod "09c179d0-e114-4d43-b296-db22061f87d6" (UID: "09c179d0-e114-4d43-b296-db22061f87d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:25:40.128916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.128842    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-istio-envoy" pod "09c179d0-e114-4d43-b296-db22061f87d6" (UID: "09c179d0-e114-4d43-b296-db22061f87d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:40.130899 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.130859    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-credential-socket" pod "09c179d0-e114-4d43-b296-db22061f87d6" (UID: "09c179d0-e114-4d43-b296-db22061f87d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:40.133723 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.133699    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c179d0-e114-4d43-b296-db22061f87d6-istio-token" pod "09c179d0-e114-4d43-b296-db22061f87d6" (UID: "09c179d0-e114-4d43-b296-db22061f87d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:40.133810 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.133738    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c179d0-e114-4d43-b296-db22061f87d6-kube-api-access-28z8w" pod "09c179d0-e114-4d43-b296-db22061f87d6" (UID: "09c179d0-e114-4d43-b296-db22061f87d6"). InnerVolumeSpecName "kube-api-access-28z8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:40.226766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.226690    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28z8w\" (UniqueName: \"kubernetes.io/projected/09c179d0-e114-4d43-b296-db22061f87d6-kube-api-access-28z8w\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.226766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.226719    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09c179d0-e114-4d43-b296-db22061f87d6-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.226766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.226729    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.226766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.226739    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09c179d0-e114-4d43-b296-db22061f87d6-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.226766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.226749    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.226766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.226757    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.226766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.226765    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.226766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.226773    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09c179d0-e114-4d43-b296-db22061f87d6-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.227134 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.226780    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09c179d0-e114-4d43-b296-db22061f87d6-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:40.406079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.406045    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"]
Oct 30 13:25:40.411039 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.411014    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bec983-openshift-default-f5ff49c7c-bdbnv"]
Oct 30 13:25:40.532626 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:40.532548    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c179d0-e114-4d43-b296-db22061f87d6" path="/var/lib/kubelet/pods/09c179d0-e114-4d43-b296-db22061f87d6/volumes"
Oct 30 13:25:41.356533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:41.356489    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-ab59dd"
Oct 30 13:25:41.454711 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:41.454678    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8-kube-api-access-qcwtq\" (UniqueName: \"kubernetes.io/projected/ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8-kube-api-access-qcwtq\") pod \"ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8\" (UID: \"ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8\") "
Oct 30 13:25:41.457990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:41.457955    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8-kube-api-access-qcwtq" pod "ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8" (UID: "ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8"). InnerVolumeSpecName "kube-api-access-qcwtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:41.556110 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:41.556067    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcwtq\" (UniqueName: \"kubernetes.io/projected/ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8-kube-api-access-qcwtq\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:42.066223 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:42.066171    5893 generic.go:356] "Generic (PLEG): container finished" podID="ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8" containerID="680f465ec27f2cc753a592ae5eae934cd806a11ddf6378d5a631e885aa656676" exitCode=137
Oct 30 13:25:42.066223 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:42.066215    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-ab59dd"
Oct 30 13:25:42.066559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:42.066456    5893 scope.go:122] "RemoveContainer" containerID="680f465ec27f2cc753a592ae5eae934cd806a11ddf6378d5a631e885aa656676"
Oct 30 13:25:42.082755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:42.082730    5893 scope.go:122] "RemoveContainer" containerID="680f465ec27f2cc753a592ae5eae934cd806a11ddf6378d5a631e885aa656676"
Oct 30 13:25:42.083135 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:25:42.083112    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"680f465ec27f2cc753a592ae5eae934cd806a11ddf6378d5a631e885aa656676\": container with ID starting with 680f465ec27f2cc753a592ae5eae934cd806a11ddf6378d5a631e885aa656676 not found: ID does not exist" containerID="680f465ec27f2cc753a592ae5eae934cd806a11ddf6378d5a631e885aa656676"
Oct 30 13:25:42.083220 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:42.083145    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"680f465ec27f2cc753a592ae5eae934cd806a11ddf6378d5a631e885aa656676"} err="failed to get container status \"680f465ec27f2cc753a592ae5eae934cd806a11ddf6378d5a631e885aa656676\": rpc error: code = NotFound desc = could not find container \"680f465ec27f2cc753a592ae5eae934cd806a11ddf6378d5a631e885aa656676\": container with ID starting with 680f465ec27f2cc753a592ae5eae934cd806a11ddf6378d5a631e885aa656676 not found: ID does not exist"
Oct 30 13:25:42.088694 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:25:42.088665    5893 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-ab59dd\" is forbidden: User \"system:node:ip-10-0-141-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-232.ec2.internal' and this object" podUID="ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8" pod="opendatahub/test-curl-9077-ab59dd"
Oct 30 13:25:42.532810 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:42.532780    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8" path="/var/lib/kubelet/pods/ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8/volumes"
Oct 30 13:25:43.076981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:43.076943    5893 generic.go:356] "Generic (PLEG): container finished" podID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerID="3b51a2e3313b390ba71d9224ab277bbb3367033ee59b1e0646a69722d0debff5" exitCode=0
Oct 30 13:25:43.077336 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:43.077003    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx" event={"ID":"daffe0a0-8e1d-4bbf-a044-67a5f3af103a","Type":"ContainerDied","Data":"3b51a2e3313b390ba71d9224ab277bbb3367033ee59b1e0646a69722d0debff5"}
Oct 30 13:25:43.077581 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:43.077482    5893 scope.go:122] "RemoveContainer" containerID="3b51a2e3313b390ba71d9224ab277bbb3367033ee59b1e0646a69722d0debff5"
Oct 30 13:25:43.079638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:43.079605    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:43.079755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:43.079689    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:43.079755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:43.079733    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:44.086724 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:44.086687    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx" event={"ID":"daffe0a0-8e1d-4bbf-a044-67a5f3af103a","Type":"ContainerStarted","Data":"f67831946436397bd927a8cd94f0f2c3f8e7bfb39fcd601c3e5b6a5966b046b2"}
Oct 30 13:25:44.362355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:44.349082    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:25:44.362355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:44.349261    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 13:25:44.362355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:44.349338    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx" podUID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 13:25:44.902593 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:44.902562    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:25:45.286286 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.286239    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"]
Oct 30 13:25:45.287348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.287325    5893 cpu_manager.go:414] "containerMap: removing container" podUID="ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8" containerName="test-curl-9077-ab59dd"
Oct 30 13:25:45.287348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.287346    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8" containerName="test-curl-9077-ab59dd"
Oct 30 13:25:45.287556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.287373    5893 cpu_manager.go:414] "containerMap: removing container" podUID="09c179d0-e114-4d43-b296-db22061f87d6" containerName="istio-proxy"
Oct 30 13:25:45.287556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.287381    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c179d0-e114-4d43-b296-db22061f87d6" containerName="istio-proxy"
Oct 30 13:25:45.287659 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.287592    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed69cbe6-afe1-46a3-be2f-9c6d40dd5ff8" containerName="test-curl-9077-ab59dd"
Oct 30 13:25:45.287659 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.287609    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c179d0-e114-4d43-b296-db22061f87d6" containerName="istio-proxy"
Oct 30 13:25:45.293971 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.292613    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.311958 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.311931    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-acd125\"/\"default-dockercfg-hfjxl\""
Oct 30 13:25:45.312209 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.312188    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-acd125\"/\"sub-test-model-acd125-kserve-self-signed-certs\""
Oct 30 13:25:45.312416 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.312388    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-home\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.312544 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.312525    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjvwr\" (UniqueName: \"kubernetes.io/projected/2912d932-69b4-407f-91a9-a18cfab1ca79-kube-api-access-tjvwr\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.313234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.312713    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2912d932-69b4-407f-91a9-a18cfab1ca79-tls-certs\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.313234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.312755    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-tmp-dir\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.313234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.312811    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-dshm\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.313234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.312879    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-model-cache\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.323601 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.323575    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-acd125\"/\"kube-root-ca.crt\""
Oct 30 13:25:45.323741 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.323679    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-acd125\"/\"openshift-service-ca.crt\""
Oct 30 13:25:45.328963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.328939    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"]
Oct 30 13:25:45.354737 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.354702    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 13:25:45.372780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.372741    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx" podUID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 13:25:45.415910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.414914    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjvwr\" (UniqueName: \"kubernetes.io/projected/2912d932-69b4-407f-91a9-a18cfab1ca79-kube-api-access-tjvwr\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.415910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.415068    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2912d932-69b4-407f-91a9-a18cfab1ca79-tls-certs\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.415910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.415095    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-tmp-dir\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.415910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.415150    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-dshm\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.415910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.415211    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-model-cache\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.415910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.415351    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-home\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.415910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.415690    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-tmp-dir\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.415910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.415725    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-home\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.416512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.416307    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-model-cache\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.418682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.418657    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-dshm\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.419003 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.418983    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2912d932-69b4-407f-91a9-a18cfab1ca79-tls-certs\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.439849 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.439814    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjvwr\" (UniqueName: \"kubernetes.io/projected/2912d932-69b4-407f-91a9-a18cfab1ca79-kube-api-access-tjvwr\") pod \"sub-test-model-acd125-kserve-56bb8887b5-4rn5x\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") " pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:45.619388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:45.619086    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:46.003776 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:46.001056    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"]
Oct 30 13:25:46.027306 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:25:46.027260    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2912d932_69b4_407f_91a9_a18cfab1ca79.slice/crio-044fe14a82ef07e0a391620e28df886dc3794d2c342a73f15fa47bd4d6693de2 WatchSource:0}: Error finding container 044fe14a82ef07e0a391620e28df886dc3794d2c342a73f15fa47bd4d6693de2: Status 404 returned error can't find the container with id 044fe14a82ef07e0a391620e28df886dc3794d2c342a73f15fa47bd4d6693de2
Oct 30 13:25:46.104664 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:46.104633    5893 generic.go:356] "Generic (PLEG): container finished" podID="bfe59aa0-d066-43f1-97de-9fa4775c71bf" containerID="c18fd38beac63be783eca55f565a520b3f5e78bede233cc24406f2f0c2626f98" exitCode=0
Oct 30 13:25:46.104855 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:46.104747    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2" event={"ID":"bfe59aa0-d066-43f1-97de-9fa4775c71bf","Type":"ContainerDied","Data":"c18fd38beac63be783eca55f565a520b3f5e78bede233cc24406f2f0c2626f98"}
Oct 30 13:25:46.106572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:46.106226    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x" event={"ID":"2912d932-69b4-407f-91a9-a18cfab1ca79","Type":"ContainerStarted","Data":"044fe14a82ef07e0a391620e28df886dc3794d2c342a73f15fa47bd4d6693de2"}
Oct 30 13:25:46.397824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:46.397574    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 13:25:46.397824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:46.397629    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx" podUID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 13:25:46.411940 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:46.411916    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"
Oct 30 13:25:46.508844 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:46.501470    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe59aa0-d066-43f1-97de-9fa4775c71bf-kube-api-access-6nbfz\" (UniqueName: \"kubernetes.io/projected/bfe59aa0-d066-43f1-97de-9fa4775c71bf-kube-api-access-6nbfz\") pod \"bfe59aa0-d066-43f1-97de-9fa4775c71bf\" (UID: \"bfe59aa0-d066-43f1-97de-9fa4775c71bf\") "
Oct 30 13:25:46.508844 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:46.502180    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe59aa0-d066-43f1-97de-9fa4775c71bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe59aa0-d066-43f1-97de-9fa4775c71bf-plugins-config-volume\") pod \"bfe59aa0-d066-43f1-97de-9fa4775c71bf\" (UID: \"bfe59aa0-d066-43f1-97de-9fa4775c71bf\") "
Oct 30 13:25:46.508844 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:46.502752    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe59aa0-d066-43f1-97de-9fa4775c71bf-plugins-config-volume" pod "bfe59aa0-d066-43f1-97de-9fa4775c71bf" (UID: "bfe59aa0-d066-43f1-97de-9fa4775c71bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:46.508844 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:46.503935    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe59aa0-d066-43f1-97de-9fa4775c71bf-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:46.512329 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:46.510337    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe59aa0-d066-43f1-97de-9fa4775c71bf-kube-api-access-6nbfz" pod "bfe59aa0-d066-43f1-97de-9fa4775c71bf" (UID: "bfe59aa0-d066-43f1-97de-9fa4775c71bf"). InnerVolumeSpecName "kube-api-access-6nbfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:46.605075 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:46.605046    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nbfz\" (UniqueName: \"kubernetes.io/projected/bfe59aa0-d066-43f1-97de-9fa4775c71bf-kube-api-access-6nbfz\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:25:46.671354 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:25:46.671118    5893 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe59aa0_d066_43f1_97de_9fa4775c71bf.slice/crio-2478f28c7445ef3d07a99c3a7bd2fb07c20c1278bae953ebe732d875638c1ee7\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe59aa0_d066_43f1_97de_9fa4775c71bf.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:25:47.120011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:47.119967    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2" event={"ID":"bfe59aa0-d066-43f1-97de-9fa4775c71bf","Type":"ContainerDied","Data":"2478f28c7445ef3d07a99c3a7bd2fb07c20c1278bae953ebe732d875638c1ee7"}
Oct 30 13:25:47.120226 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:47.120089    5893 scope.go:122] "RemoveContainer" containerID="c18fd38beac63be783eca55f565a520b3f5e78bede233cc24406f2f0c2626f98"
Oct 30 13:25:47.120372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:47.120315    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"
Oct 30 13:25:47.128029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:47.127999    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x" event={"ID":"2912d932-69b4-407f-91a9-a18cfab1ca79","Type":"ContainerStarted","Data":"ae784a2ed88d0950d94fb13f98b70af83db26e4fb537734e03eb2fbb2faa42a6"}
Oct 30 13:25:47.207403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:47.207371    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"]
Oct 30 13:25:47.218515 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:47.218487    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-619b9b-7c8755987f-kzwr2"]
Oct 30 13:25:47.270414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:47.270348    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x" podStartSLOduration=2.270322637 podStartE2EDuration="2.270322637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:47.250550637 +0000 UTC m=+2388.833980331" watchObservedRunningTime="2026-10-30 13:25:47.270322637 +0000 UTC m=+2388.853752343"
Oct 30 13:25:47.353298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:47.352687    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 13:25:47.353298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:47.352746    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx" podUID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 13:25:47.620287 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:47.620251    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:48.356300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:48.350697    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 13:25:48.356300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:48.350779    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx" podUID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 13:25:48.534717 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:48.534688    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe59aa0-d066-43f1-97de-9fa4775c71bf" path="/var/lib/kubelet/pods/bfe59aa0-d066-43f1-97de-9fa4775c71bf/volumes"
Oct 30 13:25:48.683313 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:25:48.683182    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:25:49.363300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.361852    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Oct 30 13:25:49.363300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.361909    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx" podUID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Oct 30 13:25:49.706440 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.706355    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"]
Oct 30 13:25:49.707541 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.707518    5893 cpu_manager.go:414] "containerMap: removing container" podUID="bfe59aa0-d066-43f1-97de-9fa4775c71bf" containerName="payload-pre-processing"
Oct 30 13:25:49.707690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.707542    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe59aa0-d066-43f1-97de-9fa4775c71bf" containerName="payload-pre-processing"
Oct 30 13:25:49.707801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.707783    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe59aa0-d066-43f1-97de-9fa4775c71bf" containerName="payload-pre-processing"
Oct 30 13:25:49.711795 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.711770    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:49.732221 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.732197    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-627dfa-openshift-default-dockercfg-mnpmk\""
Oct 30 13:25:49.752438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.752403    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"]
Oct 30 13:25:49.876340 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.876301    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-credential-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:49.892198 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.892154    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-workload-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:49.892588 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.892564    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-workload-certs\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:49.892868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.892849    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-token\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:49.907306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.906723    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-envoy\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:49.907306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.906783    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-data\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:49.907306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.906822    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76sxf\" (UniqueName: \"kubernetes.io/projected/29e1be4b-6574-45e6-b865-9159a3526f2f-kube-api-access-76sxf\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:49.907306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.906902    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-podinfo\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:49.907306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:49.906931    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e1be4b-6574-45e6-b865-9159a3526f2f-istiod-ca-cert\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.010066 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.009908    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-workload-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.010248 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.010082    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-workload-certs\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.010248 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.010168    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-token\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.010248 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.010233    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-envoy\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.010480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.010288    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-data\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.010480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.010324    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76sxf\" (UniqueName: \"kubernetes.io/projected/29e1be4b-6574-45e6-b865-9159a3526f2f-kube-api-access-76sxf\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.010480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.010398    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-podinfo\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.010480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.010430    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e1be4b-6574-45e6-b865-9159a3526f2f-istiod-ca-cert\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.010695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.010524    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-credential-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.011057 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.011023    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-credential-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.011599 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.011571    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-data\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.012056 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.011995    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-workload-certs\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.012207 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.012180    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-workload-socket\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.012910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.012884    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e1be4b-6574-45e6-b865-9159a3526f2f-istiod-ca-cert\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.014535 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.014510    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-podinfo\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.015396 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.015373    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-envoy\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.034589 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.034558    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76sxf\" (UniqueName: \"kubernetes.io/projected/29e1be4b-6574-45e6-b865-9159a3526f2f-kube-api-access-76sxf\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.034920 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.034901    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-token\") pod \"e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") " pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.329485 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.329399    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:50.352471 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.352445    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:25:50.354963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.354940    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:25:50.428101 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.426846    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"]
Oct 30 13:25:50.435024 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.433967    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.477468 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.477432    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-2afe58\"/\"kube-root-ca.crt\""
Oct 30 13:25:50.477653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.477598    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-2afe58\"/\"auth-test-model-2afe58-kserve-self-signed-certs\""
Oct 30 13:25:50.477714 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.477698    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-2afe58\"/\"default-dockercfg-d5gvf\""
Oct 30 13:25:50.477774 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.477761    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-2afe58\"/\"openshift-service-ca.crt\""
Oct 30 13:25:50.485788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.485701    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"]
Oct 30 13:25:50.520345 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.520252    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-home\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.520596 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.520575    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-model-cache\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.520876 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.520855    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-dshm\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.521101 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.521085    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da8cba62-e3df-4293-968a-3ce7be6b699a-tls-certs\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.521345 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.521328    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-tmp-dir\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.521485 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.521469    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp7pd\" (UniqueName: \"kubernetes.io/projected/da8cba62-e3df-4293-968a-3ce7be6b699a-kube-api-access-sp7pd\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.622941 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.622861    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp7pd\" (UniqueName: \"kubernetes.io/projected/da8cba62-e3df-4293-968a-3ce7be6b699a-kube-api-access-sp7pd\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.623087 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.623033    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-home\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.623132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.623095    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-model-cache\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.623206 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.623190    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-dshm\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.623265 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.623253    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da8cba62-e3df-4293-968a-3ce7be6b699a-tls-certs\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.623393 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.623369    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-tmp-dir\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.623606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.623514    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-home\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.623606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.623584    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-model-cache\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.623687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.623634    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-tmp-dir\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.625913 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.625892    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-dshm\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.626069 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.626052    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da8cba62-e3df-4293-968a-3ce7be6b699a-tls-certs\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.645799 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.645766    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp7pd\" (UniqueName: \"kubernetes.io/projected/da8cba62-e3df-4293-968a-3ce7be6b699a-kube-api-access-sp7pd\") pod \"auth-test-model-2afe58-kserve-564986ff9c-nfv9g\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") " pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:50.659520 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.659485    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"]
Oct 30 13:25:50.661991 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:25:50.661955    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e1be4b_6574_45e6_b865_9159a3526f2f.slice/crio-9f8a72fb89ee28f3b949675fa0b0db643c3602fe5ad1c451b22a9ebd84fb9df9 WatchSource:0}: Error finding container 9f8a72fb89ee28f3b949675fa0b0db643c3602fe5ad1c451b22a9ebd84fb9df9: Status 404 returned error can't find the container with id 9f8a72fb89ee28f3b949675fa0b0db643c3602fe5ad1c451b22a9ebd84fb9df9
Oct 30 13:25:50.666052 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.666006    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:50.666158 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.666103    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:50.666238 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.666155    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:50.754820 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:50.754748    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:51.129544 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:51.129489    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"]
Oct 30 13:25:51.132783 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:25:51.132750    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda8cba62_e3df_4293_968a_3ce7be6b699a.slice/crio-73674831f6026eec17ec54c32e5282713ab8e08ba3645a28b9c73f295df43613 WatchSource:0}: Error finding container 73674831f6026eec17ec54c32e5282713ab8e08ba3645a28b9c73f295df43613: Status 404 returned error can't find the container with id 73674831f6026eec17ec54c32e5282713ab8e08ba3645a28b9c73f295df43613
Oct 30 13:25:51.194814 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:51.194778    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj" event={"ID":"29e1be4b-6574-45e6-b865-9159a3526f2f","Type":"ContainerStarted","Data":"be35fa895f7b318db1978454e1afca757f43b2008b2e20067e19c013d2a34efe"}
Oct 30 13:25:51.194975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:51.194835    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj" event={"ID":"29e1be4b-6574-45e6-b865-9159a3526f2f","Type":"ContainerStarted","Data":"9f8a72fb89ee28f3b949675fa0b0db643c3602fe5ad1c451b22a9ebd84fb9df9"}
Oct 30 13:25:51.197661 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:51.197629    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g" event={"ID":"da8cba62-e3df-4293-968a-3ce7be6b699a","Type":"ContainerStarted","Data":"73674831f6026eec17ec54c32e5282713ab8e08ba3645a28b9c73f295df43613"}
Oct 30 13:25:51.329967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:51.329929    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:51.335859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:51.335823    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:51.387142 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:51.386721    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj" podStartSLOduration=2.386701275 podStartE2EDuration="2.386701275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:51.244000629 +0000 UTC m=+2392.827430335" watchObservedRunningTime="2026-10-30 13:25:51.386701275 +0000 UTC m=+2392.970131028"
Oct 30 13:25:52.216760 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.216634    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g" event={"ID":"da8cba62-e3df-4293-968a-3ce7be6b699a","Type":"ContainerStarted","Data":"839ece2ba1ba9cac0e77406cfb7ff85437ec02b769c6d748e3d9ab5e7773fdd3"}
Oct 30 13:25:52.224376 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.217340    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:52.225250 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.225215    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:25:52.280379 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.278146    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g" podStartSLOduration=2.278120605 podStartE2EDuration="2.278120605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:52.27278185 +0000 UTC m=+2393.856211546" watchObservedRunningTime="2026-10-30 13:25:52.278120605 +0000 UTC m=+2393.861550300"
Oct 30 13:25:52.430703 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.430670    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"]
Oct 30 13:25:52.434303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.434281    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.449318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.449265    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-e9d641-openshift-default-dockercfg-jhrsg\""
Oct 30 13:25:52.459458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.459418    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"]
Oct 30 13:25:52.472381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.469396    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-podinfo\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.472381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.469453    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-workload-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.472381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.469568    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f632f65-19d6-4073-bafc-f840bb22ecc7-istiod-ca-cert\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.472381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.469606    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-token\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.472381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.469641    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-workload-certs\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.472381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.469670    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-credential-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.472381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.469699    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65j8h\" (UniqueName: \"kubernetes.io/projected/1f632f65-19d6-4073-bafc-f840bb22ecc7-kube-api-access-65j8h\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.472381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.469864    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-data\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.472381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.469895    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-envoy\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.576442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.574964    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-data\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.576442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.575016    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-envoy\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.576442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.575235    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-podinfo\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.576442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.575288    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-workload-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.576442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.575324    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f632f65-19d6-4073-bafc-f840bb22ecc7-istiod-ca-cert\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.576442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.575351    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-token\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.576442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.575382    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-workload-certs\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.576442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.575415    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-credential-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.576442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.575440    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65j8h\" (UniqueName: \"kubernetes.io/projected/1f632f65-19d6-4073-bafc-f840bb22ecc7-kube-api-access-65j8h\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.576442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.576222    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-data\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.578827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.577941    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-workload-certs\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.578827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.578561    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-workload-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.578827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.578793    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-credential-socket\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.580093 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.580072    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f632f65-19d6-4073-bafc-f840bb22ecc7-istiod-ca-cert\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.632351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.627091    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-token\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.632351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.630587    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-envoy\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.632351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.631225    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65j8h\" (UniqueName: \"kubernetes.io/projected/1f632f65-19d6-4073-bafc-f840bb22ecc7-kube-api-access-65j8h\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.632351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.632020    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-podinfo\") pod \"e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") " pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.752646 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.752551    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:52.756840 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:52.756804    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:25:53.081449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:53.081421    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"]
Oct 30 13:25:53.082838 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:25:53.082814    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f632f65_19d6_4073_bafc_f840bb22ecc7.slice/crio-1a8e7d539fddafaf3d9bb6dc0502febd3fecf3855081c0cdae27367d74e11a00 WatchSource:0}: Error finding container 1a8e7d539fddafaf3d9bb6dc0502febd3fecf3855081c0cdae27367d74e11a00: Status 404 returned error can't find the container with id 1a8e7d539fddafaf3d9bb6dc0502febd3fecf3855081c0cdae27367d74e11a00
Oct 30 13:25:53.085886 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:53.085852    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:53.085994 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:53.085917    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:53.085994 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:53.085947    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:25:53.224437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:53.224393    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t" event={"ID":"1f632f65-19d6-4073-bafc-f840bb22ecc7","Type":"ContainerStarted","Data":"ab87ac6616b9b43c8df230fb3918f570953cbf1f474783af2aa3c621fcdafe80"}
Oct 30 13:25:53.224822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:53.224462    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t" event={"ID":"1f632f65-19d6-4073-bafc-f840bb22ecc7","Type":"ContainerStarted","Data":"1a8e7d539fddafaf3d9bb6dc0502febd3fecf3855081c0cdae27367d74e11a00"}
Oct 30 13:25:53.269605 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:53.269553    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t" podStartSLOduration=1.269535636 podStartE2EDuration="1.269535636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:53.26864843 +0000 UTC m=+2394.852078123" watchObservedRunningTime="2026-10-30 13:25:53.269535636 +0000 UTC m=+2394.852965329"
Oct 30 13:25:53.756297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:53.755352    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:54.771299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:54.766456    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 30 13:25:54.771299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:54.766518    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t" podUID="1f632f65-19d6-4073-bafc-f840bb22ecc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 30 13:25:55.619653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:55.619615    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:55.632863 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:55.632838    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:55.757972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:55.754665    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 30 13:25:55.757972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:55.754727    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t" podUID="1f632f65-19d6-4073-bafc-f840bb22ecc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 30 13:25:56.287560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:56.287523    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:25:56.767298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:56.766620    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 30 13:25:56.767298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:56.766679    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t" podUID="1f632f65-19d6-4073-bafc-f840bb22ecc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 30 13:25:57.760297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:57.759647    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 30 13:25:57.760297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:57.759705    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t" podUID="1f632f65-19d6-4073-bafc-f840bb22ecc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 30 13:25:58.760766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:58.760729    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused" start-of-body=
Oct 30 13:25:58.761812 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:58.761780    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t" podUID="1f632f65-19d6-4073-bafc-f840bb22ecc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.154:15021/healthz/ready\": dial tcp 10.134.0.154:15021: connect: connection refused"
Oct 30 13:25:59.824191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:59.824158    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:59.825557 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:59.825529    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:25:59.827023 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:25:59.826999    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:26:00.184328 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:00.184296    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"]
Oct 30 13:26:00.184559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:00.184536    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87" podUID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerName="istio-proxy" containerID="cri-o://a7069ced5288ee2af49a1220047147e8bdf11dac8c64776ba6be236c10036453" gracePeriod=30
Oct 30 13:26:00.758296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:00.758066    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:26:00.793501 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:00.793473    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:26:01.015223 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:01.015143    5893 patch_prober.go:29] interesting pod/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:26:01.015223 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:01.015202    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87" podUID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:26:01.128957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:01.128925    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:26:01.153783 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:01.153751    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:26:01.161187 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:01.161153    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:26:01.201301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:01.196405    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:26:01.332827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:01.332739    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:26:05.355814 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.355784    5893 generic.go:356] "Generic (PLEG): container finished" podID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerID="a7069ced5288ee2af49a1220047147e8bdf11dac8c64776ba6be236c10036453" exitCode=0
Oct 30 13:26:05.356318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.355857    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87" event={"ID":"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b","Type":"ContainerDied","Data":"a7069ced5288ee2af49a1220047147e8bdf11dac8c64776ba6be236c10036453"}
Oct 30 13:26:05.356318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.355902    5893 scope.go:122] "RemoveContainer" containerID="940dfc9a35b250df4ee1698effe2d55299527c217dcd71b8c60c271745b5d810"
Oct 30 13:26:05.518553 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.518524    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 13:26:05.660243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.660204    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-podinfo\") pod \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") "
Oct 30 13:26:05.660448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.660332    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istiod-ca-cert\") pod \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") "
Oct 30 13:26:05.660448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.660429    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-workload-certs\") pod \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") "
Oct 30 13:26:05.660576 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.660469    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-data\") pod \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") "
Oct 30 13:26:05.660633 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.660595    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-workload-socket\") pod \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") "
Oct 30 13:26:05.660723 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.660691    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-credential-socket\") pod \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") "
Oct 30 13:26:05.660785 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.660723    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-kube-api-access-bhzh4\" (UniqueName: \"kubernetes.io/projected/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-kube-api-access-bhzh4\") pod \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") "
Oct 30 13:26:05.660785 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.660753    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-envoy\") pod \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") "
Oct 30 13:26:05.660893 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.660782    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-workload-certs" pod "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" (UID: "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:05.660893 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.660849    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-token\" (UniqueName: \"kubernetes.io/projected/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-token\") pod \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\" (UID: \"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b\") "
Oct 30 13:26:05.660893 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.660875    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-data" pod "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" (UID: "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:05.661063 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.660984    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istiod-ca-cert" pod "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" (UID: "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:05.661640 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.661599    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-workload-socket" pod "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" (UID: "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:05.661839 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.661608    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:05.662031 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.662015    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:05.662195 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.662182    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:05.662347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.661927    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-credential-socket" pod "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" (UID: "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:05.663292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.663046    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-podinfo" pod "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" (UID: "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:26:05.663786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.663759    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-kube-api-access-bhzh4" pod "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" (UID: "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b"). InnerVolumeSpecName "kube-api-access-bhzh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:05.663884 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.663834    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-envoy" pod "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" (UID: "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:05.664188 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.664166    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-token" pod "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" (UID: "5a1b30da-f0f8-4eb0-b50b-1d2ae668801b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:05.763817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.763736    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:05.763817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.763763    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:05.763817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.763773    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:05.763817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.763782    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:05.763817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.763792    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhzh4\" (UniqueName: \"kubernetes.io/projected/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-kube-api-access-bhzh4\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:05.763817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:05.763801    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:06.366505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:06.366479    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"
Oct 30 13:26:06.366965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:06.366483    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87" event={"ID":"5a1b30da-f0f8-4eb0-b50b-1d2ae668801b","Type":"ContainerDied","Data":"822d8fb8efb73e0875ae708c2e98cafc2123b9b64a4605aedcd517ef592fa7cf"}
Oct 30 13:26:06.366965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:06.366625    5893 scope.go:122] "RemoveContainer" containerID="a7069ced5288ee2af49a1220047147e8bdf11dac8c64776ba6be236c10036453"
Oct 30 13:26:06.434961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:06.434922    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"]
Oct 30 13:26:06.446638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:06.446608    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-8b46c9-openshift-default-6c8db6467-blx87"]
Oct 30 13:26:06.532908 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:06.532874    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" path="/var/lib/kubelet/pods/5a1b30da-f0f8-4eb0-b50b-1d2ae668801b/volumes"
Oct 30 13:26:08.039326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.039289    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"]
Oct 30 13:26:08.040458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.040433    5893 cpu_manager.go:414] "containerMap: removing container" podUID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerName="istio-proxy"
Oct 30 13:26:08.040458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.040458    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerName="istio-proxy"
Oct 30 13:26:08.040730 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.040516    5893 cpu_manager.go:414] "containerMap: removing container" podUID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerName="istio-proxy"
Oct 30 13:26:08.040730 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.040522    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerName="istio-proxy"
Oct 30 13:26:08.040730 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.040689    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerName="istio-proxy"
Oct 30 13:26:08.046025 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.046004    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"]
Oct 30 13:26:08.046159 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.046133    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"
Oct 30 13:26:08.046921 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.046899    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1b30da-f0f8-4eb0-b50b-1d2ae668801b" containerName="istio-proxy"
Oct 30 13:26:08.051261 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.051103    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"
Oct 30 13:26:08.084647 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.084615    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-96ab73-dockercfg-xvpgr\""
Oct 30 13:26:08.084801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.084690    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-96ab73\""
Oct 30 13:26:08.089353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.089326    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"]
Oct 30 13:26:08.095432 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.095411    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"]
Oct 30 13:26:08.229324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.221756    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkxm8\" (UniqueName: \"kubernetes.io/projected/1df5b08d-07c1-4075-b8da-9545cabd30d9-kube-api-access-hkxm8\") pod \"payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb\" (UID: \"1df5b08d-07c1-4075-b8da-9545cabd30d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"
Oct 30 13:26:08.229324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.221998    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530acb52-18b3-4645-aad2-b5d56cc913f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph\" (UID: \"530acb52-18b3-4645-aad2-b5d56cc913f5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"
Oct 30 13:26:08.229324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.222233    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d57v\" (UniqueName: \"kubernetes.io/projected/530acb52-18b3-4645-aad2-b5d56cc913f5-kube-api-access-8d57v\") pod \"payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph\" (UID: \"530acb52-18b3-4645-aad2-b5d56cc913f5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"
Oct 30 13:26:08.229324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.222261    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df5b08d-07c1-4075-b8da-9545cabd30d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb\" (UID: \"1df5b08d-07c1-4075-b8da-9545cabd30d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"
Oct 30 13:26:08.324118 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.324030    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkxm8\" (UniqueName: \"kubernetes.io/projected/1df5b08d-07c1-4075-b8da-9545cabd30d9-kube-api-access-hkxm8\") pod \"payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb\" (UID: \"1df5b08d-07c1-4075-b8da-9545cabd30d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"
Oct 30 13:26:08.324324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.324182    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530acb52-18b3-4645-aad2-b5d56cc913f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph\" (UID: \"530acb52-18b3-4645-aad2-b5d56cc913f5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"
Oct 30 13:26:08.324401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.324387    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d57v\" (UniqueName: \"kubernetes.io/projected/530acb52-18b3-4645-aad2-b5d56cc913f5-kube-api-access-8d57v\") pod \"payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph\" (UID: \"530acb52-18b3-4645-aad2-b5d56cc913f5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"
Oct 30 13:26:08.324454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.324417    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df5b08d-07c1-4075-b8da-9545cabd30d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb\" (UID: \"1df5b08d-07c1-4075-b8da-9545cabd30d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"
Oct 30 13:26:08.325136 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.325110    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530acb52-18b3-4645-aad2-b5d56cc913f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph\" (UID: \"530acb52-18b3-4645-aad2-b5d56cc913f5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"
Oct 30 13:26:08.325201 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.325186    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df5b08d-07c1-4075-b8da-9545cabd30d9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb\" (UID: \"1df5b08d-07c1-4075-b8da-9545cabd30d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"
Oct 30 13:26:08.351041 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.350981    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkxm8\" (UniqueName: \"kubernetes.io/projected/1df5b08d-07c1-4075-b8da-9545cabd30d9-kube-api-access-hkxm8\") pod \"payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb\" (UID: \"1df5b08d-07c1-4075-b8da-9545cabd30d9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"
Oct 30 13:26:08.351537 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.351513    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d57v\" (UniqueName: \"kubernetes.io/projected/530acb52-18b3-4645-aad2-b5d56cc913f5-kube-api-access-8d57v\") pod \"payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph\" (UID: \"530acb52-18b3-4645-aad2-b5d56cc913f5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"
Oct 30 13:26:08.368661 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.368297    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"
Oct 30 13:26:08.394933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.394906    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"
Oct 30 13:26:08.694452 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.694422    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"]
Oct 30 13:26:08.695821 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:26:08.695796    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df5b08d_07c1_4075_b8da_9545cabd30d9.slice/crio-d46b31a45f6b604e64e9c4caa112351fe64f016144b4952882e9f3cc13302333 WatchSource:0}: Error finding container d46b31a45f6b604e64e9c4caa112351fe64f016144b4952882e9f3cc13302333: Status 404 returned error can't find the container with id d46b31a45f6b604e64e9c4caa112351fe64f016144b4952882e9f3cc13302333
Oct 30 13:26:08.747168 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:08.747143    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"]
Oct 30 13:26:08.752183 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:26:08.752159    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod530acb52_18b3_4645_aad2_b5d56cc913f5.slice/crio-b2583444b346b3615b3a2fa1a343ffa6a1d067d9afa76d4db684c3b7538ef779 WatchSource:0}: Error finding container b2583444b346b3615b3a2fa1a343ffa6a1d067d9afa76d4db684c3b7538ef779: Status 404 returned error can't find the container with id b2583444b346b3615b3a2fa1a343ffa6a1d067d9afa76d4db684c3b7538ef779
Oct 30 13:26:09.265177 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:09.265146    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"]
Oct 30 13:26:09.325976 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:09.325946    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"]
Oct 30 13:26:09.412663 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:09.412621    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb" event={"ID":"1df5b08d-07c1-4075-b8da-9545cabd30d9","Type":"ContainerStarted","Data":"90c7e33b7e49aa29d31971eab8bcf9ef5f7fa665ab7ffe64c47ea26a4a761ce4"}
Oct 30 13:26:09.412916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:09.412693    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb" event={"ID":"1df5b08d-07c1-4075-b8da-9545cabd30d9","Type":"ContainerStarted","Data":"d46b31a45f6b604e64e9c4caa112351fe64f016144b4952882e9f3cc13302333"}
Oct 30 13:26:09.412916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:09.412754    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"
Oct 30 13:26:09.414738 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:09.414713    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph" event={"ID":"530acb52-18b3-4645-aad2-b5d56cc913f5","Type":"ContainerStarted","Data":"95a22e88c1413d1e15322e90a48b10ac13db6ce98228084e840228cecf33da0c"}
Oct 30 13:26:09.414853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:09.414762    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph" event={"ID":"530acb52-18b3-4645-aad2-b5d56cc913f5","Type":"ContainerStarted","Data":"b2583444b346b3615b3a2fa1a343ffa6a1d067d9afa76d4db684c3b7538ef779"}
Oct 30 13:26:09.414994 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:09.414955    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"
Oct 30 13:26:09.456695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:09.456610    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb" podStartSLOduration=2.456591654 podStartE2EDuration="2.456591654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:09.453977862 +0000 UTC m=+2411.037407554" watchObservedRunningTime="2026-10-30 13:26:09.456591654 +0000 UTC m=+2411.040021354"
Oct 30 13:26:09.521247 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:09.521141    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph" podStartSLOduration=2.5211276529999997 podStartE2EDuration="2.521127653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:09.517846459 +0000 UTC m=+2411.101276152" watchObservedRunningTime="2026-10-30 13:26:09.521127653 +0000 UTC m=+2411.104557347"
Oct 30 13:26:14.266249 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:14.266204    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql" podUID="450ff90c-49e7-45c0-8e3a-6b684aedbd81" containerName="payload-processing" containerID="cri-o://078624b008fb14aa405fdd3d598eef7232cb46a2df734a3363e57d6bdd4c134c" gracePeriod=25
Oct 30 13:26:14.326364 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:14.326313    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw" podUID="f991025c-c7ba-4b0d-8676-9322d7ba6e7f" containerName="payload-pre-processing" containerID="cri-o://f3e5568e05dd8af7a349d26a7cf00e532587096ba30cb38d1c06696a8ebf4851" gracePeriod=25
Oct 30 13:26:14.455493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:14.455231    5893 generic.go:356] "Generic (PLEG): container finished" podID="450ff90c-49e7-45c0-8e3a-6b684aedbd81" containerID="078624b008fb14aa405fdd3d598eef7232cb46a2df734a3363e57d6bdd4c134c" exitCode=0
Oct 30 13:26:14.455493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:14.455420    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql" event={"ID":"450ff90c-49e7-45c0-8e3a-6b684aedbd81","Type":"ContainerDied","Data":"078624b008fb14aa405fdd3d598eef7232cb46a2df734a3363e57d6bdd4c134c"}
Oct 30 13:26:14.575364 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:14.575333    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"
Oct 30 13:26:14.617387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:14.617356    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"]
Oct 30 13:26:14.617568 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:14.617549    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy" containerID="cri-o://ae43bfa8b69daac932a13a06c9abc9a1f522d00965c60ce4573c63ebd32d2d53" gracePeriod=30
Oct 30 13:26:14.624319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:14.624292    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450ff90c-49e7-45c0-8e3a-6b684aedbd81-kube-api-access-8pxb8\" (UniqueName: \"kubernetes.io/projected/450ff90c-49e7-45c0-8e3a-6b684aedbd81-kube-api-access-8pxb8\") pod \"450ff90c-49e7-45c0-8e3a-6b684aedbd81\" (UID: \"450ff90c-49e7-45c0-8e3a-6b684aedbd81\") "
Oct 30 13:26:14.624475 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:14.624372    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450ff90c-49e7-45c0-8e3a-6b684aedbd81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450ff90c-49e7-45c0-8e3a-6b684aedbd81-plugins-config-volume\") pod \"450ff90c-49e7-45c0-8e3a-6b684aedbd81\" (UID: \"450ff90c-49e7-45c0-8e3a-6b684aedbd81\") "
Oct 30 13:26:14.624792 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:14.624764    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450ff90c-49e7-45c0-8e3a-6b684aedbd81-plugins-config-volume" pod "450ff90c-49e7-45c0-8e3a-6b684aedbd81" (UID: "450ff90c-49e7-45c0-8e3a-6b684aedbd81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:14.625253 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:14.625228    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/450ff90c-49e7-45c0-8e3a-6b684aedbd81-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:14.627806 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:14.627777    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450ff90c-49e7-45c0-8e3a-6b684aedbd81-kube-api-access-8pxb8" pod "450ff90c-49e7-45c0-8e3a-6b684aedbd81" (UID: "450ff90c-49e7-45c0-8e3a-6b684aedbd81"). InnerVolumeSpecName "kube-api-access-8pxb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:14.726897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:14.726857    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pxb8\" (UniqueName: \"kubernetes.io/projected/450ff90c-49e7-45c0-8e3a-6b684aedbd81-kube-api-access-8pxb8\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:15.465656 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:15.465622    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"
Oct 30 13:26:15.466126 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:15.465632    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql" event={"ID":"450ff90c-49e7-45c0-8e3a-6b684aedbd81","Type":"ContainerDied","Data":"b2db48117453ed9b9864d6b8c84e9f667dc5426f335991fc9afb22f57bc5d37b"}
Oct 30 13:26:15.466126 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:15.465783    5893 scope.go:122] "RemoveContainer" containerID="078624b008fb14aa405fdd3d598eef7232cb46a2df734a3363e57d6bdd4c134c"
Oct 30 13:26:15.520064 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:15.520017    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"]
Oct 30 13:26:15.523502 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:15.523477    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-60a9a6-d569c84f4-9gzql"]
Oct 30 13:26:16.533228 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:16.533195    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450ff90c-49e7-45c0-8e3a-6b684aedbd81" path="/var/lib/kubelet/pods/450ff90c-49e7-45c0-8e3a-6b684aedbd81/volumes"
Oct 30 13:26:18.483737 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:18.483701    5893 patch_prober.go:29] interesting pod/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:26:18.484152 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:18.483758    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:26:18.657139 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:26:18.657100    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:26:20.133445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.132778    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 13:26:20.296113 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.296080    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3873ae03-39a5-4f72-924d-6eba73017d74-kube-api-access-fvd2v\" (UniqueName: \"kubernetes.io/projected/3873ae03-39a5-4f72-924d-6eba73017d74-kube-api-access-fvd2v\") pod \"3873ae03-39a5-4f72-924d-6eba73017d74\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") "
Oct 30 13:26:20.298363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.298267    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-istio-envoy\") pod \"3873ae03-39a5-4f72-924d-6eba73017d74\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") "
Oct 30 13:26:20.298934 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.298915    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-workload-certs\") pod \"3873ae03-39a5-4f72-924d-6eba73017d74\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") "
Oct 30 13:26:20.299492 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.299477    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3873ae03-39a5-4f72-924d-6eba73017d74-istio-token\" (UniqueName: \"kubernetes.io/projected/3873ae03-39a5-4f72-924d-6eba73017d74-istio-token\") pod \"3873ae03-39a5-4f72-924d-6eba73017d74\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") "
Oct 30 13:26:20.300736 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.300718    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-credential-socket\") pod \"3873ae03-39a5-4f72-924d-6eba73017d74\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") "
Oct 30 13:26:20.301143 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.301108    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3873ae03-39a5-4f72-924d-6eba73017d74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3873ae03-39a5-4f72-924d-6eba73017d74-istio-podinfo\") pod \"3873ae03-39a5-4f72-924d-6eba73017d74\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") "
Oct 30 13:26:20.301965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.301943    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3873ae03-39a5-4f72-924d-6eba73017d74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3873ae03-39a5-4f72-924d-6eba73017d74-istiod-ca-cert\") pod \"3873ae03-39a5-4f72-924d-6eba73017d74\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") "
Oct 30 13:26:20.317008 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.316980    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-workload-socket\") pod \"3873ae03-39a5-4f72-924d-6eba73017d74\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") "
Oct 30 13:26:20.317385 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.317364    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-istio-data\") pod \"3873ae03-39a5-4f72-924d-6eba73017d74\" (UID: \"3873ae03-39a5-4f72-924d-6eba73017d74\") "
Oct 30 13:26:20.346213 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.299314    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-workload-certs" pod "3873ae03-39a5-4f72-924d-6eba73017d74" (UID: "3873ae03-39a5-4f72-924d-6eba73017d74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:20.346400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.301057    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-credential-socket" pod "3873ae03-39a5-4f72-924d-6eba73017d74" (UID: "3873ae03-39a5-4f72-924d-6eba73017d74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:20.346504 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.301553    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-istio-envoy" pod "3873ae03-39a5-4f72-924d-6eba73017d74" (UID: "3873ae03-39a5-4f72-924d-6eba73017d74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:20.346581 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.302447    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3873ae03-39a5-4f72-924d-6eba73017d74-istiod-ca-cert" pod "3873ae03-39a5-4f72-924d-6eba73017d74" (UID: "3873ae03-39a5-4f72-924d-6eba73017d74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:20.346657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.316812    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-workload-socket" pod "3873ae03-39a5-4f72-924d-6eba73017d74" (UID: "3873ae03-39a5-4f72-924d-6eba73017d74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:20.346734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.317747    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-istio-data" pod "3873ae03-39a5-4f72-924d-6eba73017d74" (UID: "3873ae03-39a5-4f72-924d-6eba73017d74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:26:20.360808 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.359389    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3873ae03-39a5-4f72-924d-6eba73017d74-kube-api-access-fvd2v" pod "3873ae03-39a5-4f72-924d-6eba73017d74" (UID: "3873ae03-39a5-4f72-924d-6eba73017d74"). InnerVolumeSpecName "kube-api-access-fvd2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:20.360808 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.359443    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3873ae03-39a5-4f72-924d-6eba73017d74-istio-token" pod "3873ae03-39a5-4f72-924d-6eba73017d74" (UID: "3873ae03-39a5-4f72-924d-6eba73017d74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:20.369532 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.367405    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3873ae03-39a5-4f72-924d-6eba73017d74-istio-podinfo" pod "3873ae03-39a5-4f72-924d-6eba73017d74" (UID: "3873ae03-39a5-4f72-924d-6eba73017d74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:26:20.426677 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.426648    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"
Oct 30 13:26:20.426868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.426715    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"
Oct 30 13:26:20.440121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.439932    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:20.440121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.439962    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3873ae03-39a5-4f72-924d-6eba73017d74-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:20.440121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.439977    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3873ae03-39a5-4f72-924d-6eba73017d74-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:20.440121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.439990    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:20.440121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.440005    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:20.440121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.440019    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvd2v\" (UniqueName: \"kubernetes.io/projected/3873ae03-39a5-4f72-924d-6eba73017d74-kube-api-access-fvd2v\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:20.440121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.440033    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:20.440121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.440049    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3873ae03-39a5-4f72-924d-6eba73017d74-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:20.440121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.440062    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3873ae03-39a5-4f72-924d-6eba73017d74-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:20.523526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.523486    5893 generic.go:356] "Generic (PLEG): container finished" podID="3873ae03-39a5-4f72-924d-6eba73017d74" containerID="ae43bfa8b69daac932a13a06c9abc9a1f522d00965c60ce4573c63ebd32d2d53" exitCode=0
Oct 30 13:26:20.525803 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.524439    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"
Oct 30 13:26:20.535985 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.535959    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" event={"ID":"3873ae03-39a5-4f72-924d-6eba73017d74","Type":"ContainerDied","Data":"ae43bfa8b69daac932a13a06c9abc9a1f522d00965c60ce4573c63ebd32d2d53"}
Oct 30 13:26:20.536122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.535998    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g" event={"ID":"3873ae03-39a5-4f72-924d-6eba73017d74","Type":"ContainerDied","Data":"5a358995f488744787d7438b68acf0a53d356a0563d3e338ebeb49726d412ebb"}
Oct 30 13:26:20.536122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.536014    5893 scope.go:122] "RemoveContainer" containerID="ae43bfa8b69daac932a13a06c9abc9a1f522d00965c60ce4573c63ebd32d2d53"
Oct 30 13:26:20.547406 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.547348    5893 scope.go:122] "RemoveContainer" containerID="436ec0a82c043c498497aaca1b3c15a6985f5696e4c78cbb2e4b495c8a804fde"
Oct 30 13:26:20.560778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.560756    5893 scope.go:122] "RemoveContainer" containerID="ae43bfa8b69daac932a13a06c9abc9a1f522d00965c60ce4573c63ebd32d2d53"
Oct 30 13:26:20.561191 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:26:20.561152    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae43bfa8b69daac932a13a06c9abc9a1f522d00965c60ce4573c63ebd32d2d53\": container with ID starting with ae43bfa8b69daac932a13a06c9abc9a1f522d00965c60ce4573c63ebd32d2d53 not found: ID does not exist" containerID="ae43bfa8b69daac932a13a06c9abc9a1f522d00965c60ce4573c63ebd32d2d53"
Oct 30 13:26:20.561359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.561193    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae43bfa8b69daac932a13a06c9abc9a1f522d00965c60ce4573c63ebd32d2d53"} err="failed to get container status \"ae43bfa8b69daac932a13a06c9abc9a1f522d00965c60ce4573c63ebd32d2d53\": rpc error: code = NotFound desc = could not find container \"ae43bfa8b69daac932a13a06c9abc9a1f522d00965c60ce4573c63ebd32d2d53\": container with ID starting with ae43bfa8b69daac932a13a06c9abc9a1f522d00965c60ce4573c63ebd32d2d53 not found: ID does not exist"
Oct 30 13:26:20.561359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.561219    5893 scope.go:122] "RemoveContainer" containerID="436ec0a82c043c498497aaca1b3c15a6985f5696e4c78cbb2e4b495c8a804fde"
Oct 30 13:26:20.561535 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:26:20.561510    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"436ec0a82c043c498497aaca1b3c15a6985f5696e4c78cbb2e4b495c8a804fde\": container with ID starting with 436ec0a82c043c498497aaca1b3c15a6985f5696e4c78cbb2e4b495c8a804fde not found: ID does not exist" containerID="436ec0a82c043c498497aaca1b3c15a6985f5696e4c78cbb2e4b495c8a804fde"
Oct 30 13:26:20.561594 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.561542    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436ec0a82c043c498497aaca1b3c15a6985f5696e4c78cbb2e4b495c8a804fde"} err="failed to get container status \"436ec0a82c043c498497aaca1b3c15a6985f5696e4c78cbb2e4b495c8a804fde\": rpc error: code = NotFound desc = could not find container \"436ec0a82c043c498497aaca1b3c15a6985f5696e4c78cbb2e4b495c8a804fde\": container with ID starting with 436ec0a82c043c498497aaca1b3c15a6985f5696e4c78cbb2e4b495c8a804fde not found: ID does not exist"
Oct 30 13:26:20.584566 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.584530    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"]
Oct 30 13:26:20.588702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.588662    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c2ce75-openshift-default-699569fc6c-wfm4g"]
Oct 30 13:26:20.800356 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:20.800288    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw" podUID="f991025c-c7ba-4b0d-8676-9322d7ba6e7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:26:22.540359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:22.540322    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" path="/var/lib/kubelet/pods/3873ae03-39a5-4f72-924d-6eba73017d74/volumes"
Oct 30 13:26:30.799718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:30.799675    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw" podUID="f991025c-c7ba-4b0d-8676-9322d7ba6e7f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:26:34.649065 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.649024    5893 generic.go:356] "Generic (PLEG): container finished" podID="f991025c-c7ba-4b0d-8676-9322d7ba6e7f" containerID="f3e5568e05dd8af7a349d26a7cf00e532587096ba30cb38d1c06696a8ebf4851" exitCode=0
Oct 30 13:26:34.649462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.649093    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw" event={"ID":"f991025c-c7ba-4b0d-8676-9322d7ba6e7f","Type":"ContainerDied","Data":"f3e5568e05dd8af7a349d26a7cf00e532587096ba30cb38d1c06696a8ebf4851"}
Oct 30 13:26:34.649462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.649151    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw" event={"ID":"f991025c-c7ba-4b0d-8676-9322d7ba6e7f","Type":"ContainerDied","Data":"56c5c5cb32260a574dec37bdd11a790e10d09d4bc8516b2c32c035274a0747a3"}
Oct 30 13:26:34.649462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.649168    5893 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56c5c5cb32260a574dec37bdd11a790e10d09d4bc8516b2c32c035274a0747a3"
Oct 30 13:26:34.659326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.659291    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"
Oct 30 13:26:34.771818 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.771789    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f991025c-c7ba-4b0d-8676-9322d7ba6e7f-kube-api-access-jvnsn\" (UniqueName: \"kubernetes.io/projected/f991025c-c7ba-4b0d-8676-9322d7ba6e7f-kube-api-access-jvnsn\") pod \"f991025c-c7ba-4b0d-8676-9322d7ba6e7f\" (UID: \"f991025c-c7ba-4b0d-8676-9322d7ba6e7f\") "
Oct 30 13:26:34.771818 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.771827    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f991025c-c7ba-4b0d-8676-9322d7ba6e7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f991025c-c7ba-4b0d-8676-9322d7ba6e7f-plugins-config-volume\") pod \"f991025c-c7ba-4b0d-8676-9322d7ba6e7f\" (UID: \"f991025c-c7ba-4b0d-8676-9322d7ba6e7f\") "
Oct 30 13:26:34.772317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.772295    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f991025c-c7ba-4b0d-8676-9322d7ba6e7f-plugins-config-volume" pod "f991025c-c7ba-4b0d-8676-9322d7ba6e7f" (UID: "f991025c-c7ba-4b0d-8676-9322d7ba6e7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:34.772530 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.772515    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f991025c-c7ba-4b0d-8676-9322d7ba6e7f-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:34.774030 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.774002    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f991025c-c7ba-4b0d-8676-9322d7ba6e7f-kube-api-access-jvnsn" pod "f991025c-c7ba-4b0d-8676-9322d7ba6e7f" (UID: "f991025c-c7ba-4b0d-8676-9322d7ba6e7f"). InnerVolumeSpecName "kube-api-access-jvnsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:34.820630 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.820594    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"]
Oct 30 13:26:34.821366 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.821346    5893 cpu_manager.go:414] "containerMap: removing container" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy"
Oct 30 13:26:34.821366 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.821364    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy"
Oct 30 13:26:34.821554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.821388    5893 cpu_manager.go:414] "containerMap: removing container" podUID="f991025c-c7ba-4b0d-8676-9322d7ba6e7f" containerName="payload-pre-processing"
Oct 30 13:26:34.821554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.821394    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f991025c-c7ba-4b0d-8676-9322d7ba6e7f" containerName="payload-pre-processing"
Oct 30 13:26:34.821554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.821406    5893 cpu_manager.go:414] "containerMap: removing container" podUID="450ff90c-49e7-45c0-8e3a-6b684aedbd81" containerName="payload-processing"
Oct 30 13:26:34.821554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.821411    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450ff90c-49e7-45c0-8e3a-6b684aedbd81" containerName="payload-processing"
Oct 30 13:26:34.821554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.821540    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy"
Oct 30 13:26:34.821554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.821548    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="f991025c-c7ba-4b0d-8676-9322d7ba6e7f" containerName="payload-pre-processing"
Oct 30 13:26:34.821554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.821554    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="450ff90c-49e7-45c0-8e3a-6b684aedbd81" containerName="payload-processing"
Oct 30 13:26:34.821829 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.821570    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy"
Oct 30 13:26:34.825242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.825225    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:34.843427 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.843390    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-627dfa-serving-cert\""
Oct 30 13:26:34.852747 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.852701    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"]
Oct 30 13:26:34.878284 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.878173    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtjqx\" (UniqueName: \"kubernetes.io/projected/4f7d110c-715f-4265-9e9f-542bfb750b27-kube-api-access-rtjqx\") pod \"maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:34.878463 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.878358    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:34.878463 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.878421    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d110c-715f-4265-9e9f-542bfb750b27-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:34.878595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.878499    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:34.878691 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.878673    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvnsn\" (UniqueName: \"kubernetes.io/projected/f991025c-c7ba-4b0d-8676-9322d7ba6e7f-kube-api-access-jvnsn\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:26:34.980595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.980497    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtjqx\" (UniqueName: \"kubernetes.io/projected/4f7d110c-715f-4265-9e9f-542bfb750b27-kube-api-access-rtjqx\") pod \"maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:34.980595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.980606    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:34.980595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.980659    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d110c-715f-4265-9e9f-542bfb750b27-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:34.980982 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.980760    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:34.980982 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:26:34.980903    5893 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-serving-cert: secret "maas-api-e2e-shared-b-w4-627dfa-serving-cert" not found
Oct 30 13:26:34.981103 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:26:34.980997    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-maas-api-tls podName:4f7d110c-715f-4265-9e9f-542bfb750b27 nodeName:}" failed. No retries permitted until 2026-10-30 13:26:35.480973652 +0000 UTC m=+2437.064403324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-maas-api-tls") pod "maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n" (UID: "4f7d110c-715f-4265-9e9f-542bfb750b27") : secret "maas-api-e2e-shared-b-w4-627dfa-serving-cert" not found
Oct 30 13:26:34.981463 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.981438    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d110c-715f-4265-9e9f-542bfb750b27-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:34.983305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:34.983262    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:35.029067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:35.029032    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtjqx\" (UniqueName: \"kubernetes.io/projected/4f7d110c-715f-4265-9e9f-542bfb750b27-kube-api-access-rtjqx\") pod \"maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:35.487633 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:35.487600    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:35.490520 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:35.490490    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:35.656756 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:35.656726    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"
Oct 30 13:26:35.714823 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:35.714791    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"]
Oct 30 13:26:35.724782 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:35.724755    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-60a9a6-847b54b655-mrsnw"]
Oct 30 13:26:35.744751 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:35.744662    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:36.073037 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:36.073007    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"]
Oct 30 13:26:36.074668 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:26:36.074634    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f7d110c_715f_4265_9e9f_542bfb750b27.slice/crio-62cb5f96e79ca3e027b638837be1906b4d2f39ac669d31d49bd7e10e0ea48ec4 WatchSource:0}: Error finding container 62cb5f96e79ca3e027b638837be1906b4d2f39ac669d31d49bd7e10e0ea48ec4: Status 404 returned error can't find the container with id 62cb5f96e79ca3e027b638837be1906b4d2f39ac669d31d49bd7e10e0ea48ec4
Oct 30 13:26:36.533155 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:36.533110    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f991025c-c7ba-4b0d-8676-9322d7ba6e7f" path="/var/lib/kubelet/pods/f991025c-c7ba-4b0d-8676-9322d7ba6e7f/volumes"
Oct 30 13:26:36.665400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:36.665368    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n" event={"ID":"4f7d110c-715f-4265-9e9f-542bfb750b27","Type":"ContainerStarted","Data":"56edf7b4ea7669a22d9531f07482de471fec6f4d36142f1ee7359d63db25c69d"}
Oct 30 13:26:36.665757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:36.665430    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n" event={"ID":"4f7d110c-715f-4265-9e9f-542bfb750b27","Type":"ContainerStarted","Data":"62cb5f96e79ca3e027b638837be1906b4d2f39ac669d31d49bd7e10e0ea48ec4"}
Oct 30 13:26:36.665757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:36.665506    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:36.717824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:36.717762    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n" podStartSLOduration=2.214189353 podStartE2EDuration="2.717732746s" totalImagesPullingTime="503.543393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:36.713886777 +0000 UTC m=+2438.297316471" watchObservedRunningTime="2026-10-30 13:26:36.717732746 +0000 UTC m=+2438.301162439"
Oct 30 13:26:42.681556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:42.681478    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:26:48.646959 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:26:48.646918    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:26:54.668546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:54.668511    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"]
Oct 30 13:26:54.669004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:54.668756    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85" podUID="c9a250d6-c25a-4160-8735-21bcb5a51f33" containerName="istio-proxy" containerID="cri-o://742c43ef645a6371794accbacea41bea87b8ec86c765b284d1d22d9dfa5b156b" gracePeriod=30
Oct 30 13:26:55.477553 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.477509    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"]
Oct 30 13:26:55.478750 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.478721    5893 cpu_manager.go:414] "containerMap: removing container" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy"
Oct 30 13:26:55.478750 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.478747    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3873ae03-39a5-4f72-924d-6eba73017d74" containerName="istio-proxy"
Oct 30 13:26:55.483226 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.483205    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.495425 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.495402    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"]
Oct 30 13:26:55.502129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.502107    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-bcd76c\"/\"default-dockercfg-p4hx7\""
Oct 30 13:26:55.502320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.502140    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-bcd76c\"/\"sub-test-model-bcd76c-kserve-self-signed-certs\""
Oct 30 13:26:55.511527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.511508    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-bcd76c\"/\"kube-root-ca.crt\""
Oct 30 13:26:55.511734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.511718    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-bcd76c\"/\"openshift-service-ca.crt\""
Oct 30 13:26:55.616004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.615972    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4484\" (UniqueName: \"kubernetes.io/projected/93a3e37c-f7cb-4cc4-8272-19293375dcac-kube-api-access-w4484\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.616161 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.616026    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-dshm\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.616161 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.616143    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a3e37c-f7cb-4cc4-8272-19293375dcac-tls-certs\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.616320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.616302    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-model-cache\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.616393 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.616378    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-home\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.616433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.616409    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-tmp-dir\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.718264 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.718233    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a3e37c-f7cb-4cc4-8272-19293375dcac-tls-certs\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.718712 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.718416    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-model-cache\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.718712 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.718509    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-home\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.718712 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.718545    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-tmp-dir\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.718712 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.718691    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4484\" (UniqueName: \"kubernetes.io/projected/93a3e37c-f7cb-4cc4-8272-19293375dcac-kube-api-access-w4484\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.718941 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.718750    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-dshm\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.718941 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.718775    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-model-cache\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.718941 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.718847    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-home\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.719094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.718947    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-tmp-dir\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.720955 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.720926    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-dshm\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.721092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.721015    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a3e37c-f7cb-4cc4-8272-19293375dcac-tls-certs\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.742904 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.742842    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4484\" (UniqueName: \"kubernetes.io/projected/93a3e37c-f7cb-4cc4-8272-19293375dcac-kube-api-access-w4484\") pod \"sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") " pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:55.800384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:55.800346    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:56.135749 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:56.135722    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"]
Oct 30 13:26:56.137406 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:26:56.137382    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93a3e37c_f7cb_4cc4_8272_19293375dcac.slice/crio-1dec32d38ff8680eaeff02511c3bb1cf4508669851284757280b14b02f9b2f4a WatchSource:0}: Error finding container 1dec32d38ff8680eaeff02511c3bb1cf4508669851284757280b14b02f9b2f4a: Status 404 returned error can't find the container with id 1dec32d38ff8680eaeff02511c3bb1cf4508669851284757280b14b02f9b2f4a
Oct 30 13:26:56.897809 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:56.897762    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z" event={"ID":"93a3e37c-f7cb-4cc4-8272-19293375dcac","Type":"ContainerStarted","Data":"14b1f9b88a0b7f5d243b64ef45562e145bc37dbb8b41814ae5a4e0ad65275f77"}
Oct 30 13:26:56.898208 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:56.897817    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z" event={"ID":"93a3e37c-f7cb-4cc4-8272-19293375dcac","Type":"ContainerStarted","Data":"1dec32d38ff8680eaeff02511c3bb1cf4508669851284757280b14b02f9b2f4a"}
Oct 30 13:26:56.943512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:56.943461    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z" podStartSLOduration=1.943446116 podStartE2EDuration="1.943446116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:56.941189874 +0000 UTC m=+2458.524619578" watchObservedRunningTime="2026-10-30 13:26:56.943446116 +0000 UTC m=+2458.526875808"
Oct 30 13:26:57.803041 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:57.803005    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:26:59.929651 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:59.929189    5893 generic.go:356] "Generic (PLEG): container finished" podID="c9a250d6-c25a-4160-8735-21bcb5a51f33" containerID="742c43ef645a6371794accbacea41bea87b8ec86c765b284d1d22d9dfa5b156b" exitCode=0
Oct 30 13:26:59.929651 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:26:59.929491    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85" event={"ID":"c9a250d6-c25a-4160-8735-21bcb5a51f33","Type":"ContainerDied","Data":"742c43ef645a6371794accbacea41bea87b8ec86c765b284d1d22d9dfa5b156b"}
Oct 30 13:27:00.011584 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.011541    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:27:00.070388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.065761    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a250d6-c25a-4160-8735-21bcb5a51f33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9a250d6-c25a-4160-8735-21bcb5a51f33-istiod-ca-cert\") pod \"c9a250d6-c25a-4160-8735-21bcb5a51f33\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") "
Oct 30 13:27:00.070388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.065855    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-credential-socket\") pod \"c9a250d6-c25a-4160-8735-21bcb5a51f33\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") "
Oct 30 13:27:00.070388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.065906    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-envoy\") pod \"c9a250d6-c25a-4160-8735-21bcb5a51f33\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") "
Oct 30 13:27:00.070388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.066004    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-workload-certs\") pod \"c9a250d6-c25a-4160-8735-21bcb5a51f33\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") "
Oct 30 13:27:00.070388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.066056    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-podinfo\") pod \"c9a250d6-c25a-4160-8735-21bcb5a51f33\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") "
Oct 30 13:27:00.070388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.066145    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a250d6-c25a-4160-8735-21bcb5a51f33-kube-api-access-f8whx\" (UniqueName: \"kubernetes.io/projected/c9a250d6-c25a-4160-8735-21bcb5a51f33-kube-api-access-f8whx\") pod \"c9a250d6-c25a-4160-8735-21bcb5a51f33\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") "
Oct 30 13:27:00.070388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.066259    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-token\" (UniqueName: \"kubernetes.io/projected/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-token\") pod \"c9a250d6-c25a-4160-8735-21bcb5a51f33\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") "
Oct 30 13:27:00.070388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.066354    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-data\") pod \"c9a250d6-c25a-4160-8735-21bcb5a51f33\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") "
Oct 30 13:27:00.070388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.066478    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-workload-socket\") pod \"c9a250d6-c25a-4160-8735-21bcb5a51f33\" (UID: \"c9a250d6-c25a-4160-8735-21bcb5a51f33\") "
Oct 30 13:27:00.081318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.080643    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-workload-socket" pod "c9a250d6-c25a-4160-8735-21bcb5a51f33" (UID: "c9a250d6-c25a-4160-8735-21bcb5a51f33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:00.083223 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.082883    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-workload-certs" pod "c9a250d6-c25a-4160-8735-21bcb5a51f33" (UID: "c9a250d6-c25a-4160-8735-21bcb5a51f33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:00.083223 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.083181    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-credential-socket" pod "c9a250d6-c25a-4160-8735-21bcb5a51f33" (UID: "c9a250d6-c25a-4160-8735-21bcb5a51f33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:00.083718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.083691    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-data" pod "c9a250d6-c25a-4160-8735-21bcb5a51f33" (UID: "c9a250d6-c25a-4160-8735-21bcb5a51f33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:00.088402 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.088346    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a250d6-c25a-4160-8735-21bcb5a51f33-istiod-ca-cert" pod "c9a250d6-c25a-4160-8735-21bcb5a51f33" (UID: "c9a250d6-c25a-4160-8735-21bcb5a51f33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:00.101938 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.101875    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-token" pod "c9a250d6-c25a-4160-8735-21bcb5a51f33" (UID: "c9a250d6-c25a-4160-8735-21bcb5a51f33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:00.102372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.102335    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-envoy" pod "c9a250d6-c25a-4160-8735-21bcb5a51f33" (UID: "c9a250d6-c25a-4160-8735-21bcb5a51f33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:00.106559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.106528    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-podinfo" pod "c9a250d6-c25a-4160-8735-21bcb5a51f33" (UID: "c9a250d6-c25a-4160-8735-21bcb5a51f33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:27:00.107188 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.107166    5893 scope.go:122] "RemoveContainer" containerID="f3e5568e05dd8af7a349d26a7cf00e532587096ba30cb38d1c06696a8ebf4851"
Oct 30 13:27:00.109854 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.109825    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a250d6-c25a-4160-8735-21bcb5a51f33-kube-api-access-f8whx" pod "c9a250d6-c25a-4160-8735-21bcb5a51f33" (UID: "c9a250d6-c25a-4160-8735-21bcb5a51f33"). InnerVolumeSpecName "kube-api-access-f8whx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:00.176054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.174716    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.176054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.174744    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.176054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.174759    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9a250d6-c25a-4160-8735-21bcb5a51f33-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.176054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.174773    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.176054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.174786    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.176054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.174800    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a250d6-c25a-4160-8735-21bcb5a51f33-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.176054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.174812    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.176054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.174826    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8whx\" (UniqueName: \"kubernetes.io/projected/c9a250d6-c25a-4160-8735-21bcb5a51f33-kube-api-access-f8whx\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.176054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.174842    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9a250d6-c25a-4160-8735-21bcb5a51f33-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:00.937680 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.937600    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"
Oct 30 13:27:00.937680 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.937625    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85" event={"ID":"c9a250d6-c25a-4160-8735-21bcb5a51f33","Type":"ContainerDied","Data":"1cff25e6dc67fb145a20576d12fbbe31d143c2b4641bc0e2759b5cbd543d674d"}
Oct 30 13:27:00.938130 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:00.937696    5893 scope.go:122] "RemoveContainer" containerID="742c43ef645a6371794accbacea41bea87b8ec86c765b284d1d22d9dfa5b156b"
Oct 30 13:27:01.027869 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:01.027822    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"]
Oct 30 13:27:01.039485 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:01.039457    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-60a9a6-openshift-default-dc4d6fc7c-2xj85"]
Oct 30 13:27:02.532656 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:02.532620    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9a250d6-c25a-4160-8735-21bcb5a51f33" path="/var/lib/kubelet/pods/c9a250d6-c25a-4160-8735-21bcb5a51f33/volumes"
Oct 30 13:27:02.928208 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:02.928117    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"]
Oct 30 13:27:05.807317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:05.803971    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:27:05.818378 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:05.818337    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"]
Oct 30 13:27:05.819451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:05.819426    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:27:06.016267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:06.016234    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:27:08.128342 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:08.127561    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": read tcp 10.134.0.2:56264->10.134.0.152:15021: read: connection reset by peer" start-of-body=
Oct 30 13:27:08.128342 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:08.127634    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj" podUID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": read tcp 10.134.0.2:56264->10.134.0.152:15021: read: connection reset by peer"
Oct 30 13:27:09.036147 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:09.034673    5893 generic.go:356] "Generic (PLEG): container finished" podID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerID="be35fa895f7b318db1978454e1afca757f43b2008b2e20067e19c013d2a34efe" exitCode=0
Oct 30 13:27:09.036147 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:09.034810    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj" event={"ID":"29e1be4b-6574-45e6-b865-9159a3526f2f","Type":"ContainerDied","Data":"be35fa895f7b318db1978454e1afca757f43b2008b2e20067e19c013d2a34efe"}
Oct 30 13:27:09.036147 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:09.035375    5893 scope.go:122] "RemoveContainer" containerID="be35fa895f7b318db1978454e1afca757f43b2008b2e20067e19c013d2a34efe"
Oct 30 13:27:09.038585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:09.038545    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:27:09.038732 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:09.038643    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:27:09.038792 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:09.038732    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:27:10.065298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:10.062369    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj" event={"ID":"29e1be4b-6574-45e6-b865-9159a3526f2f","Type":"ContainerStarted","Data":"7572db77ec5f5ecb7ac4fc3cc6dadf4bf39dab082a747ca3e83513b88d4cd8a5"}
Oct 30 13:27:10.330487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:10.330412    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:27:10.330636 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:10.330594    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 13:27:10.330682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:10.330639    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj" podUID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 13:27:10.823296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:10.820242    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5" podUID="c95f65d7-f870-4b93-8899-a82a15419e8e" containerName="payload-pre-processing" containerID="cri-o://2d11427e6baa82eee3c28987ddaefbb4543369a0a50cf2821516e5b6bb6ef9f9" gracePeriod=25
Oct 30 13:27:11.330100 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:11.329900    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 13:27:11.330100 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:11.329954    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj" podUID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 13:27:12.329816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:12.329776    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 13:27:12.330018 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:12.329832    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj" podUID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 13:27:13.330061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:13.330025    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 13:27:13.330504 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:13.330094    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj" podUID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 13:27:14.329819 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:14.329784    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body=
Oct 30 13:27:14.330029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:14.329843    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj" podUID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused"
Oct 30 13:27:15.334168 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:15.334142    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:27:15.334638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:15.334623    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:27:15.335305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:15.335268    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:27:15.341222 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:15.341192    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5" podUID="c95f65d7-f870-4b93-8899-a82a15419e8e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:27:17.305378 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:17.305348    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"]
Oct 30 13:27:17.977801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:17.977754    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf" podUID="00359ed8-1f53-451e-8095-a5c4cedeb18b" containerName="main" containerID="cri-o://f247fabbdd6071416554b0187ebc9269707cae72415b21ae95aaaae2e972914f" gracePeriod=45
Oct 30 13:27:18.134665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.134633    5893 generic.go:356] "Generic (PLEG): container finished" podID="00359ed8-1f53-451e-8095-a5c4cedeb18b" containerID="f247fabbdd6071416554b0187ebc9269707cae72415b21ae95aaaae2e972914f" exitCode=0
Oct 30 13:27:18.134842 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.134680    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf" event={"ID":"00359ed8-1f53-451e-8095-a5c4cedeb18b","Type":"ContainerDied","Data":"f247fabbdd6071416554b0187ebc9269707cae72415b21ae95aaaae2e972914f"}
Oct 30 13:27:18.278866 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.278837    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:27:18.394837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.394806    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00359ed8-1f53-451e-8095-a5c4cedeb18b-kube-api-access-bgmx5\" (UniqueName: \"kubernetes.io/projected/00359ed8-1f53-451e-8095-a5c4cedeb18b-kube-api-access-bgmx5\") pod \"00359ed8-1f53-451e-8095-a5c4cedeb18b\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") "
Oct 30 13:27:18.395306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.394863    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-tmp-dir\") pod \"00359ed8-1f53-451e-8095-a5c4cedeb18b\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") "
Oct 30 13:27:18.395306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.394938    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-home\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-home\") pod \"00359ed8-1f53-451e-8095-a5c4cedeb18b\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") "
Oct 30 13:27:18.395306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.395014    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-model-cache\") pod \"00359ed8-1f53-451e-8095-a5c4cedeb18b\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") "
Oct 30 13:27:18.395306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.395068    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-dshm\") pod \"00359ed8-1f53-451e-8095-a5c4cedeb18b\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") "
Oct 30 13:27:18.395306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.395178    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00359ed8-1f53-451e-8095-a5c4cedeb18b-tls-certs\" (UniqueName: \"kubernetes.io/secret/00359ed8-1f53-451e-8095-a5c4cedeb18b-tls-certs\") pod \"00359ed8-1f53-451e-8095-a5c4cedeb18b\" (UID: \"00359ed8-1f53-451e-8095-a5c4cedeb18b\") "
Oct 30 13:27:18.395306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.395189    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-tmp-dir" pod "00359ed8-1f53-451e-8095-a5c4cedeb18b" (UID: "00359ed8-1f53-451e-8095-a5c4cedeb18b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:18.395306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.395210    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-home" pod "00359ed8-1f53-451e-8095-a5c4cedeb18b" (UID: "00359ed8-1f53-451e-8095-a5c4cedeb18b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:18.395656 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.395355    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-model-cache" pod "00359ed8-1f53-451e-8095-a5c4cedeb18b" (UID: "00359ed8-1f53-451e-8095-a5c4cedeb18b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:18.395865 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.395844    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.395865 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.395866    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.396019 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.395878    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.397460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.397436    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00359ed8-1f53-451e-8095-a5c4cedeb18b-tls-certs" pod "00359ed8-1f53-451e-8095-a5c4cedeb18b" (UID: "00359ed8-1f53-451e-8095-a5c4cedeb18b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:18.397460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.397434    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00359ed8-1f53-451e-8095-a5c4cedeb18b-kube-api-access-bgmx5" pod "00359ed8-1f53-451e-8095-a5c4cedeb18b" (UID: "00359ed8-1f53-451e-8095-a5c4cedeb18b"). InnerVolumeSpecName "kube-api-access-bgmx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:18.397582 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.397528    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-dshm" pod "00359ed8-1f53-451e-8095-a5c4cedeb18b" (UID: "00359ed8-1f53-451e-8095-a5c4cedeb18b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:18.497850 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.497756    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00359ed8-1f53-451e-8095-a5c4cedeb18b-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.497850 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.497787    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgmx5\" (UniqueName: \"kubernetes.io/projected/00359ed8-1f53-451e-8095-a5c4cedeb18b-kube-api-access-bgmx5\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.497850 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:18.497800    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00359ed8-1f53-451e-8095-a5c4cedeb18b-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.649330 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:27:18.649286    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:27:19.145242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:19.145214    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"
Oct 30 13:27:19.145455 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:19.145249    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf" event={"ID":"00359ed8-1f53-451e-8095-a5c4cedeb18b","Type":"ContainerDied","Data":"4564ce3487db523bf8aae96da6889509a2bae7750b1a3ea61f1e2c2e96f9fb75"}
Oct 30 13:27:19.145455 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:19.145328    5893 scope.go:122] "RemoveContainer" containerID="f247fabbdd6071416554b0187ebc9269707cae72415b21ae95aaaae2e972914f"
Oct 30 13:27:19.201679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:19.201643    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"]
Oct 30 13:27:19.213346 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:19.213319    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2b8461/test-model-2b8461-kserve-574c5bfbb9-bj8gf"]
Oct 30 13:27:20.532734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:20.532704    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00359ed8-1f53-451e-8095-a5c4cedeb18b" path="/var/lib/kubelet/pods/00359ed8-1f53-451e-8095-a5c4cedeb18b/volumes"
Oct 30 13:27:25.341228 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:25.341191    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5" podUID="c95f65d7-f870-4b93-8899-a82a15419e8e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:27:31.129242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.129217    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"
Oct 30 13:27:31.146053 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.146023    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c95f65d7-f870-4b93-8899-a82a15419e8e-kube-api-access-g4rtj\" (UniqueName: \"kubernetes.io/projected/c95f65d7-f870-4b93-8899-a82a15419e8e-kube-api-access-g4rtj\") pod \"c95f65d7-f870-4b93-8899-a82a15419e8e\" (UID: \"c95f65d7-f870-4b93-8899-a82a15419e8e\") "
Oct 30 13:27:31.146208 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.146128    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c95f65d7-f870-4b93-8899-a82a15419e8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95f65d7-f870-4b93-8899-a82a15419e8e-plugins-config-volume\") pod \"c95f65d7-f870-4b93-8899-a82a15419e8e\" (UID: \"c95f65d7-f870-4b93-8899-a82a15419e8e\") "
Oct 30 13:27:31.146528 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.146505    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c95f65d7-f870-4b93-8899-a82a15419e8e-plugins-config-volume" pod "c95f65d7-f870-4b93-8899-a82a15419e8e" (UID: "c95f65d7-f870-4b93-8899-a82a15419e8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:31.146942 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.146921    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c95f65d7-f870-4b93-8899-a82a15419e8e-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:31.148651 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.148626    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c95f65d7-f870-4b93-8899-a82a15419e8e-kube-api-access-g4rtj" pod "c95f65d7-f870-4b93-8899-a82a15419e8e" (UID: "c95f65d7-f870-4b93-8899-a82a15419e8e"). InnerVolumeSpecName "kube-api-access-g4rtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:31.249408 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.249360    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4rtj\" (UniqueName: \"kubernetes.io/projected/c95f65d7-f870-4b93-8899-a82a15419e8e-kube-api-access-g4rtj\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:31.249602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.249431    5893 generic.go:356] "Generic (PLEG): container finished" podID="c95f65d7-f870-4b93-8899-a82a15419e8e" containerID="2d11427e6baa82eee3c28987ddaefbb4543369a0a50cf2821516e5b6bb6ef9f9" exitCode=0
Oct 30 13:27:31.249602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.249500    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5" event={"ID":"c95f65d7-f870-4b93-8899-a82a15419e8e","Type":"ContainerDied","Data":"2d11427e6baa82eee3c28987ddaefbb4543369a0a50cf2821516e5b6bb6ef9f9"}
Oct 30 13:27:31.249602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.249531    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"
Oct 30 13:27:31.249602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.249545    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5" event={"ID":"c95f65d7-f870-4b93-8899-a82a15419e8e","Type":"ContainerDied","Data":"028329e14ccdf8456729f0378fa79f8376f1c1e51ccf5acd54a526051c097ed7"}
Oct 30 13:27:31.249602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.249570    5893 scope.go:122] "RemoveContainer" containerID="2d11427e6baa82eee3c28987ddaefbb4543369a0a50cf2821516e5b6bb6ef9f9"
Oct 30 13:27:31.281315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.281256    5893 scope.go:122] "RemoveContainer" containerID="2d11427e6baa82eee3c28987ddaefbb4543369a0a50cf2821516e5b6bb6ef9f9"
Oct 30 13:27:31.281635 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:27:31.281607    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d11427e6baa82eee3c28987ddaefbb4543369a0a50cf2821516e5b6bb6ef9f9\": container with ID starting with 2d11427e6baa82eee3c28987ddaefbb4543369a0a50cf2821516e5b6bb6ef9f9 not found: ID does not exist" containerID="2d11427e6baa82eee3c28987ddaefbb4543369a0a50cf2821516e5b6bb6ef9f9"
Oct 30 13:27:31.281726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.281642    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d11427e6baa82eee3c28987ddaefbb4543369a0a50cf2821516e5b6bb6ef9f9"} err="failed to get container status \"2d11427e6baa82eee3c28987ddaefbb4543369a0a50cf2821516e5b6bb6ef9f9\": rpc error: code = NotFound desc = could not find container \"2d11427e6baa82eee3c28987ddaefbb4543369a0a50cf2821516e5b6bb6ef9f9\": container with ID starting with 2d11427e6baa82eee3c28987ddaefbb4543369a0a50cf2821516e5b6bb6ef9f9 not found: ID does not exist"
Oct 30 13:27:31.308903 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.308833    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"]
Oct 30 13:27:31.313319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:31.313294    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f3338e-7fc96f5979-gvdt5"]
Oct 30 13:27:32.329611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.329543    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68" podUID="166ae2e1-30dd-46e5-b25f-51f272d13bca" containerName="main" containerID="cri-o://2648ec65a4357eaf28a125d584831c2c9ed59fd4f2910667faa3ca5e093aa61b" gracePeriod=45
Oct 30 13:27:32.456103 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.455732    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68" podUID="166ae2e1-30dd-46e5-b25f-51f272d13bca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.145:8000/ready\": dial tcp 10.134.0.145:8000: connect: connection refused"
Oct 30 13:27:32.536480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.536447    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c95f65d7-f870-4b93-8899-a82a15419e8e" path="/var/lib/kubelet/pods/c95f65d7-f870-4b93-8899-a82a15419e8e/volumes"
Oct 30 13:27:32.638899 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.638877    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:27:32.673196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.673155    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/166ae2e1-30dd-46e5-b25f-51f272d13bca-tls-certs\" (UniqueName: \"kubernetes.io/secret/166ae2e1-30dd-46e5-b25f-51f272d13bca-tls-certs\") pod \"166ae2e1-30dd-46e5-b25f-51f272d13bca\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") "
Oct 30 13:27:32.673406 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.673255    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-dshm\") pod \"166ae2e1-30dd-46e5-b25f-51f272d13bca\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") "
Oct 30 13:27:32.673406 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.673368    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-home\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-home\") pod \"166ae2e1-30dd-46e5-b25f-51f272d13bca\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") "
Oct 30 13:27:32.673525 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.673413    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-tmp-dir\") pod \"166ae2e1-30dd-46e5-b25f-51f272d13bca\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") "
Oct 30 13:27:32.673525 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.673443    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/166ae2e1-30dd-46e5-b25f-51f272d13bca-kube-api-access-5shc2\" (UniqueName: \"kubernetes.io/projected/166ae2e1-30dd-46e5-b25f-51f272d13bca-kube-api-access-5shc2\") pod \"166ae2e1-30dd-46e5-b25f-51f272d13bca\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") "
Oct 30 13:27:32.673525 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.673509    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-model-cache\") pod \"166ae2e1-30dd-46e5-b25f-51f272d13bca\" (UID: \"166ae2e1-30dd-46e5-b25f-51f272d13bca\") "
Oct 30 13:27:32.673691 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.673665    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-home" pod "166ae2e1-30dd-46e5-b25f-51f272d13bca" (UID: "166ae2e1-30dd-46e5-b25f-51f272d13bca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:32.673792 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.673774    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-tmp-dir" pod "166ae2e1-30dd-46e5-b25f-51f272d13bca" (UID: "166ae2e1-30dd-46e5-b25f-51f272d13bca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:32.674078 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.674052    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-model-cache" pod "166ae2e1-30dd-46e5-b25f-51f272d13bca" (UID: "166ae2e1-30dd-46e5-b25f-51f272d13bca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:32.674453 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.674431    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:32.674547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.674458    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:32.674547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.674472    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:32.676055 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.675925    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-dshm" pod "166ae2e1-30dd-46e5-b25f-51f272d13bca" (UID: "166ae2e1-30dd-46e5-b25f-51f272d13bca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:32.676055 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.676019    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/166ae2e1-30dd-46e5-b25f-51f272d13bca-kube-api-access-5shc2" pod "166ae2e1-30dd-46e5-b25f-51f272d13bca" (UID: "166ae2e1-30dd-46e5-b25f-51f272d13bca"). InnerVolumeSpecName "kube-api-access-5shc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:32.676261 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.676239    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/166ae2e1-30dd-46e5-b25f-51f272d13bca-tls-certs" pod "166ae2e1-30dd-46e5-b25f-51f272d13bca" (UID: "166ae2e1-30dd-46e5-b25f-51f272d13bca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:32.776340 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.776303    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5shc2\" (UniqueName: \"kubernetes.io/projected/166ae2e1-30dd-46e5-b25f-51f272d13bca-kube-api-access-5shc2\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:32.776340 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.776339    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/166ae2e1-30dd-46e5-b25f-51f272d13bca-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:32.776610 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:32.776356    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/166ae2e1-30dd-46e5-b25f-51f272d13bca-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:33.269772 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:33.269736    5893 generic.go:356] "Generic (PLEG): container finished" podID="166ae2e1-30dd-46e5-b25f-51f272d13bca" containerID="2648ec65a4357eaf28a125d584831c2c9ed59fd4f2910667faa3ca5e093aa61b" exitCode=0
Oct 30 13:27:33.269960 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:33.269807    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"
Oct 30 13:27:33.269960 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:33.269828    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68" event={"ID":"166ae2e1-30dd-46e5-b25f-51f272d13bca","Type":"ContainerDied","Data":"2648ec65a4357eaf28a125d584831c2c9ed59fd4f2910667faa3ca5e093aa61b"}
Oct 30 13:27:33.269960 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:33.269869    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68" event={"ID":"166ae2e1-30dd-46e5-b25f-51f272d13bca","Type":"ContainerDied","Data":"fadae60a02ee30eb3e279cd94df976857a677ab16c895dcf72d59c0881e281c4"}
Oct 30 13:27:33.269960 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:33.269893    5893 scope.go:122] "RemoveContainer" containerID="2648ec65a4357eaf28a125d584831c2c9ed59fd4f2910667faa3ca5e093aa61b"
Oct 30 13:27:33.281563 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:33.280965    5893 scope.go:122] "RemoveContainer" containerID="2648ec65a4357eaf28a125d584831c2c9ed59fd4f2910667faa3ca5e093aa61b"
Oct 30 13:27:33.281814 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:27:33.281581    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2648ec65a4357eaf28a125d584831c2c9ed59fd4f2910667faa3ca5e093aa61b\": container with ID starting with 2648ec65a4357eaf28a125d584831c2c9ed59fd4f2910667faa3ca5e093aa61b not found: ID does not exist" containerID="2648ec65a4357eaf28a125d584831c2c9ed59fd4f2910667faa3ca5e093aa61b"
Oct 30 13:27:33.281814 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:33.281618    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2648ec65a4357eaf28a125d584831c2c9ed59fd4f2910667faa3ca5e093aa61b"} err="failed to get container status \"2648ec65a4357eaf28a125d584831c2c9ed59fd4f2910667faa3ca5e093aa61b\": rpc error: code = NotFound desc = could not find container \"2648ec65a4357eaf28a125d584831c2c9ed59fd4f2910667faa3ca5e093aa61b\": container with ID starting with 2648ec65a4357eaf28a125d584831c2c9ed59fd4f2910667faa3ca5e093aa61b not found: ID does not exist"
Oct 30 13:27:33.331771 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:33.331740    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"]
Oct 30 13:27:33.348382 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:33.348353    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-a5e47c/test-model-a5e47c-kserve-59ccf4d597-fjj68"]
Oct 30 13:27:34.532967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:34.532934    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="166ae2e1-30dd-46e5-b25f-51f272d13bca" path="/var/lib/kubelet/pods/166ae2e1-30dd-46e5-b25f-51f272d13bca/volumes"
Oct 30 13:27:35.219510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.219472    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"]
Oct 30 13:27:35.220263 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.220236    5893 cpu_manager.go:414] "containerMap: removing container" podUID="c9a250d6-c25a-4160-8735-21bcb5a51f33" containerName="istio-proxy"
Oct 30 13:27:35.220263 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.220262    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a250d6-c25a-4160-8735-21bcb5a51f33" containerName="istio-proxy"
Oct 30 13:27:35.220511 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.220363    5893 cpu_manager.go:414] "containerMap: removing container" podUID="166ae2e1-30dd-46e5-b25f-51f272d13bca" containerName="main"
Oct 30 13:27:35.220511 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.220373    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="166ae2e1-30dd-46e5-b25f-51f272d13bca" containerName="main"
Oct 30 13:27:35.220511 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.220395    5893 cpu_manager.go:414] "containerMap: removing container" podUID="c95f65d7-f870-4b93-8899-a82a15419e8e" containerName="payload-pre-processing"
Oct 30 13:27:35.220511 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.220401    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c95f65d7-f870-4b93-8899-a82a15419e8e" containerName="payload-pre-processing"
Oct 30 13:27:35.220511 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.220442    5893 cpu_manager.go:414] "containerMap: removing container" podUID="00359ed8-1f53-451e-8095-a5c4cedeb18b" containerName="main"
Oct 30 13:27:35.220511 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.220450    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00359ed8-1f53-451e-8095-a5c4cedeb18b" containerName="main"
Oct 30 13:27:35.220825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.220641    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="166ae2e1-30dd-46e5-b25f-51f272d13bca" containerName="main"
Oct 30 13:27:35.220825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.220657    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="c95f65d7-f870-4b93-8899-a82a15419e8e" containerName="payload-pre-processing"
Oct 30 13:27:35.220825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.220669    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="00359ed8-1f53-451e-8095-a5c4cedeb18b" containerName="main"
Oct 30 13:27:35.220825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.220679    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a250d6-c25a-4160-8735-21bcb5a51f33" containerName="istio-proxy"
Oct 30 13:27:35.225960 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.225940    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.235729 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.235704    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"]
Oct 30 13:27:35.244061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.243903    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-0e874e\"/\"kube-root-ca.crt\""
Oct 30 13:27:35.244307 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.243926    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-0e874e\"/\"openshift-service-ca.crt\""
Oct 30 13:27:35.252728 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.252703    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-0e874e\"/\"default-dockercfg-tqsdq\""
Oct 30 13:27:35.252839 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.252823    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-0e874e\"/\"sub-test-model-0e874e-kserve-self-signed-certs\""
Oct 30 13:27:35.312263 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.312219    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-dshm\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.312479 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.312299    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67a9877-0091-4783-955c-bd4e98f420c5-tls-certs\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.312479 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.312348    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-model-cache\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.312564 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.312534    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj7gn\" (UniqueName: \"kubernetes.io/projected/e67a9877-0091-4783-955c-bd4e98f420c5-kube-api-access-nj7gn\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.312628 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.312615    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-tmp-dir\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.312719 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.312708    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-home\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.414690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.414649    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj7gn\" (UniqueName: \"kubernetes.io/projected/e67a9877-0091-4783-955c-bd4e98f420c5-kube-api-access-nj7gn\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.414909 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.414752    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-tmp-dir\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.414909 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.414866    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-home\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.415054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.414921    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-dshm\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.415054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.414980    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67a9877-0091-4783-955c-bd4e98f420c5-tls-certs\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.415054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.415014    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-model-cache\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.415252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.415233    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-tmp-dir\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.415382 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.415361    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-home\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.415484 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.415453    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-model-cache\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.417442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.417423    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-dshm\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.417720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.417699    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67a9877-0091-4783-955c-bd4e98f420c5-tls-certs\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.438090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.438059    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj7gn\" (UniqueName: \"kubernetes.io/projected/e67a9877-0091-4783-955c-bd4e98f420c5-kube-api-access-nj7gn\") pod \"sub-test-model-0e874e-kserve-786d99979f-vfq48\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") " pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.546448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.546355    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:35.884901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:35.884823    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"]
Oct 30 13:27:35.893321 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:27:35.893291    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode67a9877_0091_4783_955c_bd4e98f420c5.slice/crio-bb5350f4ea95212d9e8f281b9b33fa95b1b52e7fb58607ee784466f6a1b63cee WatchSource:0}: Error finding container bb5350f4ea95212d9e8f281b9b33fa95b1b52e7fb58607ee784466f6a1b63cee: Status 404 returned error can't find the container with id bb5350f4ea95212d9e8f281b9b33fa95b1b52e7fb58607ee784466f6a1b63cee
Oct 30 13:27:36.296358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:36.296313    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48" event={"ID":"e67a9877-0091-4783-955c-bd4e98f420c5","Type":"ContainerStarted","Data":"1129ef404e0b58d0a4185016da1029c10a3ebcd6ce044885eb8a3d5afce70eec"}
Oct 30 13:27:36.296540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:36.296372    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48" event={"ID":"e67a9877-0091-4783-955c-bd4e98f420c5","Type":"ContainerStarted","Data":"bb5350f4ea95212d9e8f281b9b33fa95b1b52e7fb58607ee784466f6a1b63cee"}
Oct 30 13:27:36.339114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:36.339056    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48" podStartSLOduration=1.339043326 podStartE2EDuration="1.339043326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:36.337232789 +0000 UTC m=+2497.920662502" watchObservedRunningTime="2026-10-30 13:27:36.339043326 +0000 UTC m=+2497.922473018"
Oct 30 13:27:36.547185 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:36.547097    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:38.937187 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:38.937149    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"]
Oct 30 13:27:38.943435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:38.943409    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:38.972972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:38.969379    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-8a5d5e-openshift-default-dockercfg-t586t\""
Oct 30 13:27:38.974132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:38.974102    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"]
Oct 30 13:27:39.092299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.086066    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.092299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.086132    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlfw7\" (UniqueName: \"kubernetes.io/projected/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-kube-api-access-mlfw7\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.092299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.086171    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.092299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.086201    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.092299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.086313    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.092299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.086353    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.092299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.086412    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.092299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.086452    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.092299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.086546    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.190146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.187962    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.190146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.188075    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.190146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.188241    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.190146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.188312    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlfw7\" (UniqueName: \"kubernetes.io/projected/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-kube-api-access-mlfw7\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.190146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.188351    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.190146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.188380    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.190146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.188468    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.190146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.188507    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.190146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.188565    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.190146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.189428    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.190146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.189753    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.190146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.190051    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.190146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.190126    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.191012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.190425    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.195559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.195517    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.199755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.199725    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.217033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.217001    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlfw7\" (UniqueName: \"kubernetes.io/projected/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-kube-api-access-mlfw7\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.218913 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.218883    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.262969 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.262940    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:39.602473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.602434    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"]
Oct 30 13:27:39.614387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.614344    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:27:39.614488 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.614434    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:27:39.614488 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:39.614472    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:27:40.333469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:40.333431    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql" event={"ID":"721c5b17-8eeb-4e8b-94b4-2d46a18a384c","Type":"ContainerStarted","Data":"fc9e6ffdc69314a52f0184a5f4dc0faa824e7f7d0642a236f71620c0051ede81"}
Oct 30 13:27:40.333933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:40.333496    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql" event={"ID":"721c5b17-8eeb-4e8b-94b4-2d46a18a384c","Type":"ContainerStarted","Data":"260e6f0bbf4dc92e3621742e2023f567e8a4e9e4cf66b7e1cc9743d6858fa40f"}
Oct 30 13:27:40.382918 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:40.382864    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql" podStartSLOduration=2.3828449640000002 podStartE2EDuration="2.382844964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:40.379700683 +0000 UTC m=+2501.963130386" watchObservedRunningTime="2026-10-30 13:27:40.382844964 +0000 UTC m=+2501.966274659"
Oct 30 13:27:41.263423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:41.263387    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:41.269820 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:41.269790    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:41.341020 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:41.340991    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:41.342028 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:41.342008    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:27:45.547782 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:45.547747    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:45.566472 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:45.566436    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:46.393564 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:46.393516    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:27:48.647303 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:27:48.647251    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:27:52.683575 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:52.683540    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"]
Oct 30 13:27:52.684129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:52.683779    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" containerName="istio-proxy" containerID="cri-o://de1b3149dc338493d154c351ab5aa362fdbf4f6be8fa5190202bc7ccf6baab37" gracePeriod=30
Oct 30 13:27:57.969867 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:57.969839    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:27:58.043074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.043040    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-workload-socket\") pod \"26375037-b209-4e8f-8dec-821f2a597ffb\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") "
Oct 30 13:27:58.043301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.043098    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26375037-b209-4e8f-8dec-821f2a597ffb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26375037-b209-4e8f-8dec-821f2a597ffb-istio-podinfo\") pod \"26375037-b209-4e8f-8dec-821f2a597ffb\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") "
Oct 30 13:27:58.043301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.043136    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-credential-socket\") pod \"26375037-b209-4e8f-8dec-821f2a597ffb\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") "
Oct 30 13:27:58.043301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.043183    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-istio-envoy\") pod \"26375037-b209-4e8f-8dec-821f2a597ffb\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") "
Oct 30 13:27:58.043301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.043206    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-istio-data\") pod \"26375037-b209-4e8f-8dec-821f2a597ffb\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") "
Oct 30 13:27:58.043559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.043383    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26375037-b209-4e8f-8dec-821f2a597ffb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26375037-b209-4e8f-8dec-821f2a597ffb-istiod-ca-cert\") pod \"26375037-b209-4e8f-8dec-821f2a597ffb\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") "
Oct 30 13:27:58.043559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.043521    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-workload-certs\") pod \"26375037-b209-4e8f-8dec-821f2a597ffb\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") "
Oct 30 13:27:58.043669 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.043607    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26375037-b209-4e8f-8dec-821f2a597ffb-kube-api-access-vhk5s\" (UniqueName: \"kubernetes.io/projected/26375037-b209-4e8f-8dec-821f2a597ffb-kube-api-access-vhk5s\") pod \"26375037-b209-4e8f-8dec-821f2a597ffb\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") "
Oct 30 13:27:58.043669 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.043637    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26375037-b209-4e8f-8dec-821f2a597ffb-istio-token\" (UniqueName: \"kubernetes.io/projected/26375037-b209-4e8f-8dec-821f2a597ffb-istio-token\") pod \"26375037-b209-4e8f-8dec-821f2a597ffb\" (UID: \"26375037-b209-4e8f-8dec-821f2a597ffb\") "
Oct 30 13:27:58.047195 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.046031    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-istio-data" pod "26375037-b209-4e8f-8dec-821f2a597ffb" (UID: "26375037-b209-4e8f-8dec-821f2a597ffb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:58.047195 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.046139    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-workload-certs" pod "26375037-b209-4e8f-8dec-821f2a597ffb" (UID: "26375037-b209-4e8f-8dec-821f2a597ffb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:58.047195 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.046461    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26375037-b209-4e8f-8dec-821f2a597ffb-istiod-ca-cert" pod "26375037-b209-4e8f-8dec-821f2a597ffb" (UID: "26375037-b209-4e8f-8dec-821f2a597ffb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:58.047195 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.047135    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-credential-socket" pod "26375037-b209-4e8f-8dec-821f2a597ffb" (UID: "26375037-b209-4e8f-8dec-821f2a597ffb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:58.047870 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.047849    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-workload-socket" pod "26375037-b209-4e8f-8dec-821f2a597ffb" (UID: "26375037-b209-4e8f-8dec-821f2a597ffb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:58.053311 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.051708    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26375037-b209-4e8f-8dec-821f2a597ffb-istio-token" pod "26375037-b209-4e8f-8dec-821f2a597ffb" (UID: "26375037-b209-4e8f-8dec-821f2a597ffb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:58.057700 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.057648    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-istio-envoy" pod "26375037-b209-4e8f-8dec-821f2a597ffb" (UID: "26375037-b209-4e8f-8dec-821f2a597ffb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:27:58.057807 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.057666    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26375037-b209-4e8f-8dec-821f2a597ffb-istio-podinfo" pod "26375037-b209-4e8f-8dec-821f2a597ffb" (UID: "26375037-b209-4e8f-8dec-821f2a597ffb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:27:58.060330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.059681    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26375037-b209-4e8f-8dec-821f2a597ffb-kube-api-access-vhk5s" pod "26375037-b209-4e8f-8dec-821f2a597ffb" (UID: "26375037-b209-4e8f-8dec-821f2a597ffb"). InnerVolumeSpecName "kube-api-access-vhk5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:58.145758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.145715    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.145758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.145749    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26375037-b209-4e8f-8dec-821f2a597ffb-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.145758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.145763    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.146018 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.145777    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.146018 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.145789    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.146018 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.145801    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26375037-b209-4e8f-8dec-821f2a597ffb-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.146018 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.145810    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26375037-b209-4e8f-8dec-821f2a597ffb-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.146018 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.145821    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhk5s\" (UniqueName: \"kubernetes.io/projected/26375037-b209-4e8f-8dec-821f2a597ffb-kube-api-access-vhk5s\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.146018 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.145833    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26375037-b209-4e8f-8dec-821f2a597ffb-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:27:58.537304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.537246    5893 generic.go:356] "Generic (PLEG): container finished" podID="26375037-b209-4e8f-8dec-821f2a597ffb" containerID="de1b3149dc338493d154c351ab5aa362fdbf4f6be8fa5190202bc7ccf6baab37" exitCode=0
Oct 30 13:27:58.537461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.537303    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" event={"ID":"26375037-b209-4e8f-8dec-821f2a597ffb","Type":"ContainerDied","Data":"de1b3149dc338493d154c351ab5aa362fdbf4f6be8fa5190202bc7ccf6baab37"}
Oct 30 13:27:58.537461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.537337    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"
Oct 30 13:27:58.537461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.537365    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk" event={"ID":"26375037-b209-4e8f-8dec-821f2a597ffb","Type":"ContainerDied","Data":"aa21329cce5aa388637740982b66aafb4b30709843f08c020d3b98ff0367da28"}
Oct 30 13:27:58.537461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.537395    5893 scope.go:122] "RemoveContainer" containerID="de1b3149dc338493d154c351ab5aa362fdbf4f6be8fa5190202bc7ccf6baab37"
Oct 30 13:27:58.550407 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.550384    5893 scope.go:122] "RemoveContainer" containerID="3097b6426a10e870bbd3eaf255bec1bda747cfbff9f37057ce164ab8cdac4e6f"
Oct 30 13:27:58.563163 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.563138    5893 scope.go:122] "RemoveContainer" containerID="de1b3149dc338493d154c351ab5aa362fdbf4f6be8fa5190202bc7ccf6baab37"
Oct 30 13:27:58.563554 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:27:58.563523    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1b3149dc338493d154c351ab5aa362fdbf4f6be8fa5190202bc7ccf6baab37\": container with ID starting with de1b3149dc338493d154c351ab5aa362fdbf4f6be8fa5190202bc7ccf6baab37 not found: ID does not exist" containerID="de1b3149dc338493d154c351ab5aa362fdbf4f6be8fa5190202bc7ccf6baab37"
Oct 30 13:27:58.563654 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.563565    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1b3149dc338493d154c351ab5aa362fdbf4f6be8fa5190202bc7ccf6baab37"} err="failed to get container status \"de1b3149dc338493d154c351ab5aa362fdbf4f6be8fa5190202bc7ccf6baab37\": rpc error: code = NotFound desc = could not find container \"de1b3149dc338493d154c351ab5aa362fdbf4f6be8fa5190202bc7ccf6baab37\": container with ID starting with de1b3149dc338493d154c351ab5aa362fdbf4f6be8fa5190202bc7ccf6baab37 not found: ID does not exist"
Oct 30 13:27:58.563654 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.563589    5893 scope.go:122] "RemoveContainer" containerID="3097b6426a10e870bbd3eaf255bec1bda747cfbff9f37057ce164ab8cdac4e6f"
Oct 30 13:27:58.563889 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:27:58.563867    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3097b6426a10e870bbd3eaf255bec1bda747cfbff9f37057ce164ab8cdac4e6f\": container with ID starting with 3097b6426a10e870bbd3eaf255bec1bda747cfbff9f37057ce164ab8cdac4e6f not found: ID does not exist" containerID="3097b6426a10e870bbd3eaf255bec1bda747cfbff9f37057ce164ab8cdac4e6f"
Oct 30 13:27:58.563940 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.563900    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3097b6426a10e870bbd3eaf255bec1bda747cfbff9f37057ce164ab8cdac4e6f"} err="failed to get container status \"3097b6426a10e870bbd3eaf255bec1bda747cfbff9f37057ce164ab8cdac4e6f\": rpc error: code = NotFound desc = could not find container \"3097b6426a10e870bbd3eaf255bec1bda747cfbff9f37057ce164ab8cdac4e6f\": container with ID starting with 3097b6426a10e870bbd3eaf255bec1bda747cfbff9f37057ce164ab8cdac4e6f not found: ID does not exist"
Oct 30 13:27:58.601789 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.601760    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"]
Oct 30 13:27:58.612886 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:27:58.612866    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2b8461-openshift-default-7c444f8877-4bwmk"]
Oct 30 13:28:00.532776 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:00.532740    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" path="/var/lib/kubelet/pods/26375037-b209-4e8f-8dec-821f2a597ffb/volumes"
Oct 30 13:28:18.646741 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:28:18.646698    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:28:20.615433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.615400    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"]
Oct 30 13:28:20.616302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.616284    5893 cpu_manager.go:414] "containerMap: removing container" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" containerName="istio-proxy"
Oct 30 13:28:20.616378 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.616303    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" containerName="istio-proxy"
Oct 30 13:28:20.616378 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.616328    5893 cpu_manager.go:414] "containerMap: removing container" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" containerName="istio-proxy"
Oct 30 13:28:20.616378 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.616333    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" containerName="istio-proxy"
Oct 30 13:28:20.616473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.616433    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" containerName="istio-proxy"
Oct 30 13:28:20.616473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.616446    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="26375037-b209-4e8f-8dec-821f2a597ffb" containerName="istio-proxy"
Oct 30 13:28:20.621875 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.621855    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:20.656341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.648163    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-8a5d5e-serving-cert\""
Oct 30 13:28:20.656341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.651783    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"]
Oct 30 13:28:20.719752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.719716    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25b93077-be04-4210-9cd9-c4b1f182a1d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:20.719934 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.719786    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b93077-be04-4210-9cd9-c4b1f182a1d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:20.719986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.719944    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbq49\" (UniqueName: \"kubernetes.io/projected/25b93077-be04-4210-9cd9-c4b1f182a1d1-kube-api-access-bbq49\") pod \"maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:20.720072 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.720056    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b93077-be04-4210-9cd9-c4b1f182a1d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:20.821654 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.821621    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbq49\" (UniqueName: \"kubernetes.io/projected/25b93077-be04-4210-9cd9-c4b1f182a1d1-kube-api-access-bbq49\") pod \"maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:20.821847 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.821715    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b93077-be04-4210-9cd9-c4b1f182a1d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:20.821908 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.821841    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25b93077-be04-4210-9cd9-c4b1f182a1d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:20.821975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.821907    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b93077-be04-4210-9cd9-c4b1f182a1d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:20.822441 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.822415    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b93077-be04-4210-9cd9-c4b1f182a1d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:20.824478 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.824449    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b93077-be04-4210-9cd9-c4b1f182a1d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:20.824597 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.824515    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25b93077-be04-4210-9cd9-c4b1f182a1d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:20.846029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.845963    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbq49\" (UniqueName: \"kubernetes.io/projected/25b93077-be04-4210-9cd9-c4b1f182a1d1-kube-api-access-bbq49\") pod \"maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:20.937047 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:20.936958    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:21.267384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:21.267353    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"]
Oct 30 13:28:21.269759 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:28:21.269732    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b93077_be04_4210_9cd9_c4b1f182a1d1.slice/crio-84cd1eef75e62a71efe9d2f23057cf0ddd69546a0828e9b4d550a47c2680d46f WatchSource:0}: Error finding container 84cd1eef75e62a71efe9d2f23057cf0ddd69546a0828e9b4d550a47c2680d46f: Status 404 returned error can't find the container with id 84cd1eef75e62a71efe9d2f23057cf0ddd69546a0828e9b4d550a47c2680d46f
Oct 30 13:28:21.725020 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:21.724967    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64" event={"ID":"25b93077-be04-4210-9cd9-c4b1f182a1d1","Type":"ContainerStarted","Data":"b7be9ed899d2d72567f3aa3dd5db096ccae666765f159b9998eb86b289d19e25"}
Oct 30 13:28:21.725480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:21.725044    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64" event={"ID":"25b93077-be04-4210-9cd9-c4b1f182a1d1","Type":"ContainerStarted","Data":"84cd1eef75e62a71efe9d2f23057cf0ddd69546a0828e9b4d550a47c2680d46f"}
Oct 30 13:28:21.725480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:21.725210    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:21.767413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:21.767355    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64" podStartSLOduration=1.449916719 podStartE2EDuration="1.767334216s" totalImagesPullingTime="317.417497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:21.765842317 +0000 UTC m=+2543.349272022" watchObservedRunningTime="2026-10-30 13:28:21.767334216 +0000 UTC m=+2543.350763911"
Oct 30 13:28:27.737657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:27.737628    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:28:29.891863 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:29.891829    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"]
Oct 30 13:28:29.892516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:29.892042    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89" podUID="5a4c9c9c-a0dd-4f38-979f-bc5567660a5f" containerName="maas-api" containerID="cri-o://f0d5f1ca355ad55936305d184d2f810ef9ebbf60bdb1a99bf2e84c043540dcc4" gracePeriod=30
Oct 30 13:28:30.222927 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.222897    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:28:30.287729 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.287693    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-kube-api-access-lb52j\" (UniqueName: \"kubernetes.io/projected/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-kube-api-access-lb52j\") pod \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") "
Oct 30 13:28:30.287936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.287833    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-metrics-certs\") pod \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") "
Oct 30 13:28:30.288101 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.288078    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-openshift-service-ca\") pod \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") "
Oct 30 13:28:30.288241 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.288148    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-maas-api-tls\") pod \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\" (UID: \"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f\") "
Oct 30 13:28:30.290576 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.288699    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-openshift-service-ca" pod "5a4c9c9c-a0dd-4f38-979f-bc5567660a5f" (UID: "5a4c9c9c-a0dd-4f38-979f-bc5567660a5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:30.290576 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.289286    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.291016 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.290986    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-maas-api-tls" pod "5a4c9c9c-a0dd-4f38-979f-bc5567660a5f" (UID: "5a4c9c9c-a0dd-4f38-979f-bc5567660a5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:30.292092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.292067    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-kube-api-access-lb52j" pod "5a4c9c9c-a0dd-4f38-979f-bc5567660a5f" (UID: "5a4c9c9c-a0dd-4f38-979f-bc5567660a5f"). InnerVolumeSpecName "kube-api-access-lb52j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:30.293002 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.292981    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-metrics-certs" pod "5a4c9c9c-a0dd-4f38-979f-bc5567660a5f" (UID: "5a4c9c9c-a0dd-4f38-979f-bc5567660a5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:30.392539 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.392495    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.392786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.392769    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb52j\" (UniqueName: \"kubernetes.io/projected/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-kube-api-access-lb52j\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.393170 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.393147    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.797565 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.797469    5893 generic.go:356] "Generic (PLEG): container finished" podID="5a4c9c9c-a0dd-4f38-979f-bc5567660a5f" containerID="f0d5f1ca355ad55936305d184d2f810ef9ebbf60bdb1a99bf2e84c043540dcc4" exitCode=0
Oct 30 13:28:30.797565 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.797528    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"
Oct 30 13:28:30.797798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.797532    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89" event={"ID":"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f","Type":"ContainerDied","Data":"f0d5f1ca355ad55936305d184d2f810ef9ebbf60bdb1a99bf2e84c043540dcc4"}
Oct 30 13:28:30.797798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.797649    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89" event={"ID":"5a4c9c9c-a0dd-4f38-979f-bc5567660a5f","Type":"ContainerDied","Data":"1817d48b882d4cada67720da4ca0ef324b5ed255cde8d5e7f8a54d545e266784"}
Oct 30 13:28:30.797798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.797680    5893 scope.go:122] "RemoveContainer" containerID="f0d5f1ca355ad55936305d184d2f810ef9ebbf60bdb1a99bf2e84c043540dcc4"
Oct 30 13:28:30.824124 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.824103    5893 scope.go:122] "RemoveContainer" containerID="f0d5f1ca355ad55936305d184d2f810ef9ebbf60bdb1a99bf2e84c043540dcc4"
Oct 30 13:28:30.824497 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:28:30.824476    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d5f1ca355ad55936305d184d2f810ef9ebbf60bdb1a99bf2e84c043540dcc4\": container with ID starting with f0d5f1ca355ad55936305d184d2f810ef9ebbf60bdb1a99bf2e84c043540dcc4 not found: ID does not exist" containerID="f0d5f1ca355ad55936305d184d2f810ef9ebbf60bdb1a99bf2e84c043540dcc4"
Oct 30 13:28:30.824585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.824511    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d5f1ca355ad55936305d184d2f810ef9ebbf60bdb1a99bf2e84c043540dcc4"} err="failed to get container status \"f0d5f1ca355ad55936305d184d2f810ef9ebbf60bdb1a99bf2e84c043540dcc4\": rpc error: code = NotFound desc = could not find container \"f0d5f1ca355ad55936305d184d2f810ef9ebbf60bdb1a99bf2e84c043540dcc4\": container with ID starting with f0d5f1ca355ad55936305d184d2f810ef9ebbf60bdb1a99bf2e84c043540dcc4 not found: ID does not exist"
Oct 30 13:28:30.852869 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.852836    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"]
Oct 30 13:28:30.860001 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:30.859969    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a5e47c-8649c79f95-8pl89"]
Oct 30 13:28:32.351544 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.347564    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"]
Oct 30 13:28:32.351544 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.348604    5893 cpu_manager.go:414] "containerMap: removing container" podUID="5a4c9c9c-a0dd-4f38-979f-bc5567660a5f" containerName="maas-api"
Oct 30 13:28:32.351544 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.348623    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a4c9c9c-a0dd-4f38-979f-bc5567660a5f" containerName="maas-api"
Oct 30 13:28:32.351544 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.348825    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a4c9c9c-a0dd-4f38-979f-bc5567660a5f" containerName="maas-api"
Oct 30 13:28:32.357012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.353429    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.384804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.384775    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-627dfa\"/\"openshift-service-ca.crt\""
Oct 30 13:28:32.385016 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.384877    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-627dfa\"/\"kube-root-ca.crt\""
Oct 30 13:28:32.385262 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.385247    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-627dfa\"/\"default-dockercfg-5f5xw\""
Oct 30 13:28:32.385544 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.385527    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-627dfa\"/\"auth-test-model-627dfa-kserve-self-signed-certs\""
Oct 30 13:28:32.390021 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.389996    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"]
Oct 30 13:28:32.434750 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.434715    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-dshm\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.434937 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.434834    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-tmp-dir\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.434937 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.434879    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-tls-certs\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.435120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.435099    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-home\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.435216 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.435198    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-model-cache\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.435324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.435240    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r9xx\" (UniqueName: \"kubernetes.io/projected/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-kube-api-access-7r9xx\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.533503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.533465    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a4c9c9c-a0dd-4f38-979f-bc5567660a5f" path="/var/lib/kubelet/pods/5a4c9c9c-a0dd-4f38-979f-bc5567660a5f/volumes"
Oct 30 13:28:32.536427 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.536401    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-home\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.536595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.536467    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-model-cache\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.536595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.536491    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r9xx\" (UniqueName: \"kubernetes.io/projected/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-kube-api-access-7r9xx\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.536595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.536523    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-dshm\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.536595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.536570    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-tmp-dir\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.536844 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.536612    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-tls-certs\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.536914 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.536892    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-home\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.536968 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.536927    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-model-cache\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.537012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.536967    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-tmp-dir\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.539057 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.539037    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-dshm\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.539549 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.539227    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-tls-certs\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.571525 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.571493    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r9xx\" (UniqueName: \"kubernetes.io/projected/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-kube-api-access-7r9xx\") pod \"auth-test-model-627dfa-kserve-86944d78b9-5cxch\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") " pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:32.679586 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:32.679491    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:33.031824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:33.031781    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"]
Oct 30 13:28:33.032116 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:28:33.032091    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78dce1e2_01b4_42d2_bed5_7daeac16ffa9.slice/crio-9670ac4968850d4823a0559e1a867e9f61e6f32e283bb7d605897c80aed4aaae WatchSource:0}: Error finding container 9670ac4968850d4823a0559e1a867e9f61e6f32e283bb7d605897c80aed4aaae: Status 404 returned error can't find the container with id 9670ac4968850d4823a0559e1a867e9f61e6f32e283bb7d605897c80aed4aaae
Oct 30 13:28:33.821194 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:33.821155    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch" event={"ID":"78dce1e2-01b4-42d2-bed5-7daeac16ffa9","Type":"ContainerStarted","Data":"f5f251e0f16e6cc2d7705e03cbdee652681c41d2c51046223549870be2476678"}
Oct 30 13:28:33.821765 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:33.821203    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch" event={"ID":"78dce1e2-01b4-42d2-bed5-7daeac16ffa9","Type":"ContainerStarted","Data":"9670ac4968850d4823a0559e1a867e9f61e6f32e283bb7d605897c80aed4aaae"}
Oct 30 13:28:33.866462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:33.866392    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch" podStartSLOduration=1.8663713739999999 podStartE2EDuration="1.866371374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:33.863682575 +0000 UTC m=+2555.447112285" watchObservedRunningTime="2026-10-30 13:28:33.866371374 +0000 UTC m=+2555.449801068"
Oct 30 13:28:34.680570 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:34.680521    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:38.569745 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:38.569710    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"]
Oct 30 13:28:38.570250 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:38.569953    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44" podUID="fb104279-c853-4843-a0f7-e55eb170c620" containerName="istio-proxy" containerID="cri-o://255da8addcadf2572c0ad665014a5163d16ee9a523a99044e79362d90510fdb8" gracePeriod=30
Oct 30 13:28:39.128047 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.128006    5893 patch_prober.go:29] interesting pod/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:28:39.128329 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.128086    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44" podUID="fb104279-c853-4843-a0f7-e55eb170c620" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:28:39.811038 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.811003    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"]
Oct 30 13:28:39.816073 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.816046    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.830947 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.830922    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-1ba901-openshift-default-dockercfg-fl7pd\""
Oct 30 13:28:39.846854 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.846821    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"]
Oct 30 13:28:39.863259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.863219    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.863641 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.863613    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m964k\" (UniqueName: \"kubernetes.io/projected/64c1d503-512b-4521-af3f-668e94ed5b99-kube-api-access-m964k\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.863824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.863796    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.863984 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.863966    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.864103 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.864088    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c1d503-512b-4521-af3f-668e94ed5b99-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.864237 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.864222    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.864438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.864410    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.864563 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.864458    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c1d503-512b-4521-af3f-668e94ed5b99-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.864563 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.864488    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c1d503-512b-4521-af3f-668e94ed5b99-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.965825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.965779    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.966009 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.965894    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m964k\" (UniqueName: \"kubernetes.io/projected/64c1d503-512b-4521-af3f-668e94ed5b99-kube-api-access-m964k\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.966009 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.965957    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.966122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.966011    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.966122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.966038    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c1d503-512b-4521-af3f-668e94ed5b99-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.966122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.966082    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.966290 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.966166    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.966290 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.966204    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c1d503-512b-4521-af3f-668e94ed5b99-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.966290 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.966233    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c1d503-512b-4521-af3f-668e94ed5b99-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.966462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.966292    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.966462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.966410    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.966577 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.966477    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.966632 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.966585    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.967210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.967181    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c1d503-512b-4521-af3f-668e94ed5b99-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.968970 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.968949    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:39.969387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:39.969364    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c1d503-512b-4521-af3f-668e94ed5b99-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:40.003076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:40.003042    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c1d503-512b-4521-af3f-668e94ed5b99-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:40.003319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:40.003259    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m964k\" (UniqueName: \"kubernetes.io/projected/64c1d503-512b-4521-af3f-668e94ed5b99-kube-api-access-m964k\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:40.134135 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:40.134046    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:40.490929 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:40.490882    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"]
Oct 30 13:28:40.493750 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:28:40.493700    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c1d503_512b_4521_af3f_668e94ed5b99.slice/crio-68389413d0431cdcf850fa9b7bb3c2f2d249e2b647c0e51631aa6702e250ca53 WatchSource:0}: Error finding container 68389413d0431cdcf850fa9b7bb3c2f2d249e2b647c0e51631aa6702e250ca53: Status 404 returned error can't find the container with id 68389413d0431cdcf850fa9b7bb3c2f2d249e2b647c0e51631aa6702e250ca53
Oct 30 13:28:40.496750 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:40.496714    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:28:40.496861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:40.496786    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:28:40.496861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:40.496819    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:28:40.886766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:40.886657    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" event={"ID":"64c1d503-512b-4521-af3f-668e94ed5b99","Type":"ContainerStarted","Data":"214caddcae79781a68b6918a19ba6a299bf15c42a3f346f857ba5868515234e5"}
Oct 30 13:28:40.886766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:40.886722    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" event={"ID":"64c1d503-512b-4521-af3f-668e94ed5b99","Type":"ContainerStarted","Data":"68389413d0431cdcf850fa9b7bb3c2f2d249e2b647c0e51631aa6702e250ca53"}
Oct 30 13:28:40.963305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:40.962402    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" podStartSLOduration=1.962383763 podStartE2EDuration="1.962383763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:40.945824345 +0000 UTC m=+2562.529254039" watchObservedRunningTime="2026-10-30 13:28:40.962383763 +0000 UTC m=+2562.545813458"
Oct 30 13:28:41.134433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:41.134404    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:41.140141 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:41.140083    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:41.893685 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:41.893653    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:41.894660 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:41.894636    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:28:42.680546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:42.680511    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:42.693862 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:42.693836    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:42.911012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:42.910983    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:28:43.912255 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:43.912217    5893 generic.go:356] "Generic (PLEG): container finished" podID="fb104279-c853-4843-a0f7-e55eb170c620" containerID="255da8addcadf2572c0ad665014a5163d16ee9a523a99044e79362d90510fdb8" exitCode=0
Oct 30 13:28:43.912695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:43.912322    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44" event={"ID":"fb104279-c853-4843-a0f7-e55eb170c620","Type":"ContainerDied","Data":"255da8addcadf2572c0ad665014a5163d16ee9a523a99044e79362d90510fdb8"}
Oct 30 13:28:43.912695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:43.912386    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44" event={"ID":"fb104279-c853-4843-a0f7-e55eb170c620","Type":"ContainerDied","Data":"b04078a1261f9b42895a89a0329d7eaead77fed59bfd0144eba69258d9d398f9"}
Oct 30 13:28:43.912695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:43.912417    5893 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b04078a1261f9b42895a89a0329d7eaead77fed59bfd0144eba69258d9d398f9"
Oct 30 13:28:43.928647 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:43.928616    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:28:44.030474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.030429    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb104279-c853-4843-a0f7-e55eb170c620-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb104279-c853-4843-a0f7-e55eb170c620-istio-podinfo\") pod \"fb104279-c853-4843-a0f7-e55eb170c620\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") "
Oct 30 13:28:44.030694 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.030580    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb104279-c853-4843-a0f7-e55eb170c620-kube-api-access-68x7g\" (UniqueName: \"kubernetes.io/projected/fb104279-c853-4843-a0f7-e55eb170c620-kube-api-access-68x7g\") pod \"fb104279-c853-4843-a0f7-e55eb170c620\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") "
Oct 30 13:28:44.030798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.030748    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-workload-certs\") pod \"fb104279-c853-4843-a0f7-e55eb170c620\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") "
Oct 30 13:28:44.030863 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.030813    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-istio-envoy\") pod \"fb104279-c853-4843-a0f7-e55eb170c620\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") "
Oct 30 13:28:44.030863 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.030853    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb104279-c853-4843-a0f7-e55eb170c620-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb104279-c853-4843-a0f7-e55eb170c620-istiod-ca-cert\") pod \"fb104279-c853-4843-a0f7-e55eb170c620\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") "
Oct 30 13:28:44.030969 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.030892    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-istio-data\") pod \"fb104279-c853-4843-a0f7-e55eb170c620\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") "
Oct 30 13:28:44.030969 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.030933    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-workload-socket\") pod \"fb104279-c853-4843-a0f7-e55eb170c620\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") "
Oct 30 13:28:44.031087 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.030983    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-credential-socket\") pod \"fb104279-c853-4843-a0f7-e55eb170c620\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") "
Oct 30 13:28:44.031087 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.031046    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb104279-c853-4843-a0f7-e55eb170c620-istio-token\" (UniqueName: \"kubernetes.io/projected/fb104279-c853-4843-a0f7-e55eb170c620-istio-token\") pod \"fb104279-c853-4843-a0f7-e55eb170c620\" (UID: \"fb104279-c853-4843-a0f7-e55eb170c620\") "
Oct 30 13:28:44.031997 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.031376    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb104279-c853-4843-a0f7-e55eb170c620-istiod-ca-cert" pod "fb104279-c853-4843-a0f7-e55eb170c620" (UID: "fb104279-c853-4843-a0f7-e55eb170c620"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:44.031997 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.031940    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-credential-socket" pod "fb104279-c853-4843-a0f7-e55eb170c620" (UID: "fb104279-c853-4843-a0f7-e55eb170c620"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:44.031997 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.031956    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-workload-socket" pod "fb104279-c853-4843-a0f7-e55eb170c620" (UID: "fb104279-c853-4843-a0f7-e55eb170c620"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:44.031997 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.031969    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb104279-c853-4843-a0f7-e55eb170c620-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.032372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.032331    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-istio-data" pod "fb104279-c853-4843-a0f7-e55eb170c620" (UID: "fb104279-c853-4843-a0f7-e55eb170c620"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:44.032526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.032491    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-workload-certs" pod "fb104279-c853-4843-a0f7-e55eb170c620" (UID: "fb104279-c853-4843-a0f7-e55eb170c620"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:44.034740 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.034706    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb104279-c853-4843-a0f7-e55eb170c620-istio-podinfo" pod "fb104279-c853-4843-a0f7-e55eb170c620" (UID: "fb104279-c853-4843-a0f7-e55eb170c620"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:28:44.035203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.035167    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-istio-envoy" pod "fb104279-c853-4843-a0f7-e55eb170c620" (UID: "fb104279-c853-4843-a0f7-e55eb170c620"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:44.035328 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.035251    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb104279-c853-4843-a0f7-e55eb170c620-kube-api-access-68x7g" pod "fb104279-c853-4843-a0f7-e55eb170c620" (UID: "fb104279-c853-4843-a0f7-e55eb170c620"). InnerVolumeSpecName "kube-api-access-68x7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:44.035576 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.035552    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb104279-c853-4843-a0f7-e55eb170c620-istio-token" pod "fb104279-c853-4843-a0f7-e55eb170c620" (UID: "fb104279-c853-4843-a0f7-e55eb170c620"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:44.133122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.133087    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68x7g\" (UniqueName: \"kubernetes.io/projected/fb104279-c853-4843-a0f7-e55eb170c620-kube-api-access-68x7g\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.133122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.133119    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.133122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.133130    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.133445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.133138    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.133445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.133146    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.133445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.133154    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb104279-c853-4843-a0f7-e55eb170c620-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.133445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.133163    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb104279-c853-4843-a0f7-e55eb170c620-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.133445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.133171    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb104279-c853-4843-a0f7-e55eb170c620-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:28:44.919633 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.919553    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"
Oct 30 13:28:44.989940 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.989908    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"]
Oct 30 13:28:44.998332 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:44.998296    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a5e47c-openshift-default-66d7979c75-75k44"]
Oct 30 13:28:46.531994 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:46.531965    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb104279-c853-4843-a0f7-e55eb170c620" path="/var/lib/kubelet/pods/fb104279-c853-4843-a0f7-e55eb170c620/volumes"
Oct 30 13:28:48.682910 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:28:48.682849    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:28:54.262938 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.262895    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"]
Oct 30 13:28:54.263733 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.263713    5893 cpu_manager.go:414] "containerMap: removing container" podUID="fb104279-c853-4843-a0f7-e55eb170c620" containerName="istio-proxy"
Oct 30 13:28:54.263733 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.263734    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb104279-c853-4843-a0f7-e55eb170c620" containerName="istio-proxy"
Oct 30 13:28:54.263976 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.263927    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb104279-c853-4843-a0f7-e55eb170c620" containerName="istio-proxy"
Oct 30 13:28:54.279455 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.279421    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.319588 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.319558    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"]
Oct 30 13:28:54.330962 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.330929    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.331128 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.331099    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.331183 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.331150    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.331373 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.331343    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.331491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.331474    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.331552 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.331529    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6wv\" (UniqueName: \"kubernetes.io/projected/805ec71c-0b0c-48a9-a154-7fbf568f73ba-kube-api-access-vc6wv\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.331606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.331558    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.331689 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.331662    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.331816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.331709    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.433586 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.433553    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.433794 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.433593    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.433794 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.433666    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.433794 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.433726    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.433794 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.433789    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6wv\" (UniqueName: \"kubernetes.io/projected/805ec71c-0b0c-48a9-a154-7fbf568f73ba-kube-api-access-vc6wv\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.434038 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.433816    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.434038 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.433844    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.434038 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.433875    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.434038 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.434017    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.434520 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.434493    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-workload-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.434995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.434496    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istiod-ca-cert\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.435149 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.434531    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-data\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.435149 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.434783    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-workload-certs\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.435149 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.434931    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-credential-socket\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.436761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.436739    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-envoy\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.437124 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.437108    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-podinfo\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.457770 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.457743    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-token\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.463701 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.463676    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6wv\" (UniqueName: \"kubernetes.io/projected/805ec71c-0b0c-48a9-a154-7fbf568f73ba-kube-api-access-vc6wv\") pod \"e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") " pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:54.638985 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:54.638909    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:55.003918 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:55.003872    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"]
Oct 30 13:28:55.010116 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:28:55.010086    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805ec71c_0b0c_48a9_a154_7fbf568f73ba.slice/crio-d23cafc125802c919ba71ce6c9dd4a3dc8d59505106b12c757aa7226aba667aa WatchSource:0}: Error finding container d23cafc125802c919ba71ce6c9dd4a3dc8d59505106b12c757aa7226aba667aa: Status 404 returned error can't find the container with id d23cafc125802c919ba71ce6c9dd4a3dc8d59505106b12c757aa7226aba667aa
Oct 30 13:28:55.013738 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:55.013703    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:28:55.013845 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:55.013777    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:28:55.013845 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:55.013808    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:28:55.049232 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:55.049196    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj" event={"ID":"805ec71c-0b0c-48a9-a154-7fbf568f73ba","Type":"ContainerStarted","Data":"d23cafc125802c919ba71ce6c9dd4a3dc8d59505106b12c757aa7226aba667aa"}
Oct 30 13:28:56.075012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:56.074970    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj" event={"ID":"805ec71c-0b0c-48a9-a154-7fbf568f73ba","Type":"ContainerStarted","Data":"cc5765085692b031c63afac12f8909598aab3f7a7fc7486109de03abad941fcd"}
Oct 30 13:28:56.076929 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:56.076902    5893 generic.go:356] "Generic (PLEG): container finished" podID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerID="fc9e6ffdc69314a52f0184a5f4dc0faa824e7f7d0642a236f71620c0051ede81" exitCode=0
Oct 30 13:28:56.077074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:56.076958    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql" event={"ID":"721c5b17-8eeb-4e8b-94b4-2d46a18a384c","Type":"ContainerDied","Data":"fc9e6ffdc69314a52f0184a5f4dc0faa824e7f7d0642a236f71620c0051ede81"}
Oct 30 13:28:56.077330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:56.077315    5893 scope.go:122] "RemoveContainer" containerID="fc9e6ffdc69314a52f0184a5f4dc0faa824e7f7d0642a236f71620c0051ede81"
Oct 30 13:28:56.079189 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:56.079154    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:28:56.079310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:56.079285    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:28:56.079434 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:56.079338    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:28:56.133646 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:56.133599    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj" podStartSLOduration=2.133583208 podStartE2EDuration="2.133583208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:56.132374814 +0000 UTC m=+2577.715804519" watchObservedRunningTime="2026-10-30 13:28:56.133583208 +0000 UTC m=+2577.717012953"
Oct 30 13:28:56.353362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:56.350367    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:28:56.639366 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:56.639261    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:28:56.660297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:56.659848    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 13:28:56.660297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:56.659923    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj" podUID="805ec71c-0b0c-48a9-a154-7fbf568f73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 13:28:57.105318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:57.104503    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql" event={"ID":"721c5b17-8eeb-4e8b-94b4-2d46a18a384c","Type":"ContainerStarted","Data":"be190cf826b3356ece2e753fe3e0efc84bfd8139859a57b0765d2ceb1b78ed76"}
Oct 30 13:28:57.263680 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:57.263648    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:28:57.275456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:57.275417    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 13:28:57.275747 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:57.275721    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql" podUID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 13:28:57.640166 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:57.640135    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 13:28:57.640415 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:57.640184    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj" podUID="805ec71c-0b0c-48a9-a154-7fbf568f73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 13:28:58.268521 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.268062    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 13:28:58.268521 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.268117    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql" podUID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 13:28:58.295298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.285773    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"]
Oct 30 13:28:58.310789 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.310750    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"]
Oct 30 13:28:58.310980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.310907    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.328713 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.328456    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-0b2c92-openshift-default-dockercfg-4kbg9\""
Oct 30 13:28:58.430773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.430745    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-workload-certs\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.430919 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.430827    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-podinfo\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.430919 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.430877    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-workload-socket\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.430998 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.430930    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istiod-ca-cert\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.431073 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.431052    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqsbh\" (UniqueName: \"kubernetes.io/projected/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-kube-api-access-xqsbh\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.431146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.431130    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-token\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.431219 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.431207    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-envoy\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.431415 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.431402    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-data\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.431459 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.431434    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-credential-socket\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.533140 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.533061    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-data\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.533140 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.533105    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-credential-socket\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.533351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.533153    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-workload-certs\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.533351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.533198    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-podinfo\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.533351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.533226    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-workload-socket\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.533351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.533242    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istiod-ca-cert\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.533351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.533291    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqsbh\" (UniqueName: \"kubernetes.io/projected/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-kube-api-access-xqsbh\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.533351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.533309    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-token\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.533545 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.533353    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-envoy\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.534768 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.534737    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-credential-socket\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.535004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.534979    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-data\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.535787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.535597    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istiod-ca-cert\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.536295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.536205    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-workload-socket\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.536881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.536865    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-workload-certs\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.537533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.537341    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-podinfo\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.537644 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.537555    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-envoy\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.582298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.582041    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqsbh\" (UniqueName: \"kubernetes.io/projected/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-kube-api-access-xqsbh\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.604299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.599772    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-token\") pod \"e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") " pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:58.639401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.639365    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 13:28:58.639575 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.639428    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj" podUID="805ec71c-0b0c-48a9-a154-7fbf568f73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 13:28:58.719919 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.719889    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-0b2c92-openshift-default-dockercfg-4kbg9\""
Oct 30 13:28:58.726662 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:58.726636    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:28:59.198608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:59.198529    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"]
Oct 30 13:28:59.204297 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:28:59.204253    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8525f4ea_caea_4321_8c2e_4c6fa4fe06da.slice/crio-75f1828568104c242b76625abbf5742aee1b1d7aa1ff6b42f55b8c9c3384190f WatchSource:0}: Error finding container 75f1828568104c242b76625abbf5742aee1b1d7aa1ff6b42f55b8c9c3384190f: Status 404 returned error can't find the container with id 75f1828568104c242b76625abbf5742aee1b1d7aa1ff6b42f55b8c9c3384190f
Oct 30 13:28:59.207804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:59.207767    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:28:59.207894 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:59.207860    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:28:59.207936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:59.207904    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:28:59.267532 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:59.265485    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 13:28:59.267532 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:59.265551    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql" podUID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 13:28:59.639722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:59.639650    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 13:28:59.639722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:28:59.639709    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj" podUID="805ec71c-0b0c-48a9-a154-7fbf568f73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 13:29:00.129330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:00.129266    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln" event={"ID":"8525f4ea-caea-4321-8c2e-4c6fa4fe06da","Type":"ContainerStarted","Data":"610b534d7f749ae4444f821a6513c40fdbb0a774cd3e09093c088aac338228f7"}
Oct 30 13:29:00.129585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:00.129568    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln" event={"ID":"8525f4ea-caea-4321-8c2e-4c6fa4fe06da","Type":"ContainerStarted","Data":"75f1828568104c242b76625abbf5742aee1b1d7aa1ff6b42f55b8c9c3384190f"}
Oct 30 13:29:00.267299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:00.266381    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 13:29:00.267299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:00.266442    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql" podUID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 13:29:00.640392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:00.640359    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Oct 30 13:29:00.641004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:00.640978    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj" podUID="805ec71c-0b0c-48a9-a154-7fbf568f73ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Oct 30 13:29:00.731718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:00.731682    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:29:00.735648 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:00.735617    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 13:29:00.735800 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:00.735682    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln" podUID="8525f4ea-caea-4321-8c2e-4c6fa4fe06da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 13:29:01.265793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:01.265750    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 13:29:01.268417 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:01.268379    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql" podUID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 13:29:01.738021 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:01.729120    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 13:29:01.738021 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:01.729178    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln" podUID="8525f4ea-caea-4321-8c2e-4c6fa4fe06da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 13:29:02.118862 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:02.118738    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln" podStartSLOduration=4.118713351 podStartE2EDuration="4.118713351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:00.207540331 +0000 UTC m=+2581.790970025" watchObservedRunningTime="2026-10-30 13:29:02.118713351 +0000 UTC m=+2583.702143048"
Oct 30 13:29:02.124464 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:02.124424    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"]
Oct 30 13:29:02.275695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:02.275422    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Oct 30 13:29:02.275695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:02.275498    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql" podUID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Oct 30 13:29:02.409571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:02.409105    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:29:02.430636 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:02.429755    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:29:02.434189 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:02.431708    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:29:02.601887 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:02.601853    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"]
Oct 30 13:29:02.736932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:02.731438    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 13:29:02.736932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:02.731497    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln" podUID="8525f4ea-caea-4321-8c2e-4c6fa4fe06da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 13:29:03.279351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:03.279313    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:29:03.279871 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:03.279766    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:29:03.280734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:03.280711    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:29:03.727443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:03.727414    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 13:29:03.727616 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:03.727467    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln" podUID="8525f4ea-caea-4321-8c2e-4c6fa4fe06da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 13:29:04.727321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:04.727258    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 13:29:04.727744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:04.727363    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln" podUID="8525f4ea-caea-4321-8c2e-4c6fa4fe06da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 13:29:05.727220 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:05.727185    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body=
Oct 30 13:29:05.727454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:05.727237    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln" podUID="8525f4ea-caea-4321-8c2e-4c6fa4fe06da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused"
Oct 30 13:29:06.785958 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:06.785311    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:29:06.785958 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:06.785751    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:29:06.788494 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:06.788462    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:29:17.516773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:17.516724    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x" podUID="2912d932-69b4-407f-91a9-a18cfab1ca79" containerName="main" containerID="cri-o://ae784a2ed88d0950d94fb13f98b70af83db26e4fb537734e03eb2fbb2faa42a6" gracePeriod=45
Oct 30 13:29:17.660561 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:17.660516    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g" podUID="da8cba62-e3df-4293-968a-3ce7be6b699a" containerName="main" containerID="cri-o://839ece2ba1ba9cac0e77406cfb7ff85437ec02b769c6d748e3d9ab5e7773fdd3" gracePeriod=45
Oct 30 13:29:18.145309 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.144175    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:29:18.251747 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.251679    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2912d932-69b4-407f-91a9-a18cfab1ca79-tls-certs\" (UniqueName: \"kubernetes.io/secret/2912d932-69b4-407f-91a9-a18cfab1ca79-tls-certs\") pod \"2912d932-69b4-407f-91a9-a18cfab1ca79\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") "
Oct 30 13:29:18.251747 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.251736    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2912d932-69b4-407f-91a9-a18cfab1ca79-kube-api-access-tjvwr\" (UniqueName: \"kubernetes.io/projected/2912d932-69b4-407f-91a9-a18cfab1ca79-kube-api-access-tjvwr\") pod \"2912d932-69b4-407f-91a9-a18cfab1ca79\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") "
Oct 30 13:29:18.251973 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.251891    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-home\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-home\") pod \"2912d932-69b4-407f-91a9-a18cfab1ca79\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") "
Oct 30 13:29:18.252037 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.251979    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-model-cache\") pod \"2912d932-69b4-407f-91a9-a18cfab1ca79\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") "
Oct 30 13:29:18.252219 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.252193    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-dshm\") pod \"2912d932-69b4-407f-91a9-a18cfab1ca79\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") "
Oct 30 13:29:18.252363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.252293    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-tmp-dir\") pod \"2912d932-69b4-407f-91a9-a18cfab1ca79\" (UID: \"2912d932-69b4-407f-91a9-a18cfab1ca79\") "
Oct 30 13:29:18.252657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.252519    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-home" pod "2912d932-69b4-407f-91a9-a18cfab1ca79" (UID: "2912d932-69b4-407f-91a9-a18cfab1ca79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:18.253174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.252753    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-tmp-dir" pod "2912d932-69b4-407f-91a9-a18cfab1ca79" (UID: "2912d932-69b4-407f-91a9-a18cfab1ca79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:18.253174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.252791    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-model-cache" pod "2912d932-69b4-407f-91a9-a18cfab1ca79" (UID: "2912d932-69b4-407f-91a9-a18cfab1ca79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:18.253174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.253103    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:29:18.253174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.253124    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:29:18.253174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.253152    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:29:18.255332 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.255249    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2912d932-69b4-407f-91a9-a18cfab1ca79-kube-api-access-tjvwr" pod "2912d932-69b4-407f-91a9-a18cfab1ca79" (UID: "2912d932-69b4-407f-91a9-a18cfab1ca79"). InnerVolumeSpecName "kube-api-access-tjvwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:18.256132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.256104    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2912d932-69b4-407f-91a9-a18cfab1ca79-tls-certs" pod "2912d932-69b4-407f-91a9-a18cfab1ca79" (UID: "2912d932-69b4-407f-91a9-a18cfab1ca79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:18.256507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.256481    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-dshm" pod "2912d932-69b4-407f-91a9-a18cfab1ca79" (UID: "2912d932-69b4-407f-91a9-a18cfab1ca79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:18.264040 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.264010    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:29:18.354527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.354427    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-dshm\") pod \"da8cba62-e3df-4293-968a-3ce7be6b699a\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") "
Oct 30 13:29:18.354709 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.354550    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-home\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-home\") pod \"da8cba62-e3df-4293-968a-3ce7be6b699a\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") "
Oct 30 13:29:18.354709 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.354639    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da8cba62-e3df-4293-968a-3ce7be6b699a-tls-certs\" (UniqueName: \"kubernetes.io/secret/da8cba62-e3df-4293-968a-3ce7be6b699a-tls-certs\") pod \"da8cba62-e3df-4293-968a-3ce7be6b699a\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") "
Oct 30 13:29:18.354838 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.354750    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-model-cache\") pod \"da8cba62-e3df-4293-968a-3ce7be6b699a\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") "
Oct 30 13:29:18.354838 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.354784    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da8cba62-e3df-4293-968a-3ce7be6b699a-kube-api-access-sp7pd\" (UniqueName: \"kubernetes.io/projected/da8cba62-e3df-4293-968a-3ce7be6b699a-kube-api-access-sp7pd\") pod \"da8cba62-e3df-4293-968a-3ce7be6b699a\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") "
Oct 30 13:29:18.354921 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.354839    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-home" pod "da8cba62-e3df-4293-968a-3ce7be6b699a" (UID: "da8cba62-e3df-4293-968a-3ce7be6b699a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:18.355009 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.354986    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-model-cache" pod "da8cba62-e3df-4293-968a-3ce7be6b699a" (UID: "da8cba62-e3df-4293-968a-3ce7be6b699a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:18.355009 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.355000    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-tmp-dir\") pod \"da8cba62-e3df-4293-968a-3ce7be6b699a\" (UID: \"da8cba62-e3df-4293-968a-3ce7be6b699a\") "
Oct 30 13:29:18.355197 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.355181    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-tmp-dir" pod "da8cba62-e3df-4293-968a-3ce7be6b699a" (UID: "da8cba62-e3df-4293-968a-3ce7be6b699a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:18.356084 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.356059    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:29:18.356084 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.356086    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2912d932-69b4-407f-91a9-a18cfab1ca79-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:29:18.356226 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.356100    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2912d932-69b4-407f-91a9-a18cfab1ca79-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:29:18.356226 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.356115    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjvwr\" (UniqueName: \"kubernetes.io/projected/2912d932-69b4-407f-91a9-a18cfab1ca79-kube-api-access-tjvwr\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:29:18.356226 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.356129    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:29:18.356226 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.356143    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:29:18.356928 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.356903    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-dshm" pod "da8cba62-e3df-4293-968a-3ce7be6b699a" (UID: "da8cba62-e3df-4293-968a-3ce7be6b699a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:18.357159 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.357137    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da8cba62-e3df-4293-968a-3ce7be6b699a-tls-certs" pod "da8cba62-e3df-4293-968a-3ce7be6b699a" (UID: "da8cba62-e3df-4293-968a-3ce7be6b699a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:18.357411 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.357391    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da8cba62-e3df-4293-968a-3ce7be6b699a-kube-api-access-sp7pd" pod "da8cba62-e3df-4293-968a-3ce7be6b699a" (UID: "da8cba62-e3df-4293-968a-3ce7be6b699a"). InnerVolumeSpecName "kube-api-access-sp7pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:18.457429 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.457402    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da8cba62-e3df-4293-968a-3ce7be6b699a-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:29:18.457429 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.457428    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da8cba62-e3df-4293-968a-3ce7be6b699a-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:29:18.457631 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.457440    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp7pd\" (UniqueName: \"kubernetes.io/projected/da8cba62-e3df-4293-968a-3ce7be6b699a-kube-api-access-sp7pd\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:29:18.647581 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:29:18.647483    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:29:18.703132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.703101    5893 generic.go:356] "Generic (PLEG): container finished" podID="da8cba62-e3df-4293-968a-3ce7be6b699a" containerID="839ece2ba1ba9cac0e77406cfb7ff85437ec02b769c6d748e3d9ab5e7773fdd3" exitCode=0
Oct 30 13:29:18.703301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.703143    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"
Oct 30 13:29:18.703301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.703178    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g" event={"ID":"da8cba62-e3df-4293-968a-3ce7be6b699a","Type":"ContainerDied","Data":"839ece2ba1ba9cac0e77406cfb7ff85437ec02b769c6d748e3d9ab5e7773fdd3"}
Oct 30 13:29:18.703301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.703223    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g" event={"ID":"da8cba62-e3df-4293-968a-3ce7be6b699a","Type":"ContainerDied","Data":"73674831f6026eec17ec54c32e5282713ab8e08ba3645a28b9c73f295df43613"}
Oct 30 13:29:18.703301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.703246    5893 scope.go:122] "RemoveContainer" containerID="839ece2ba1ba9cac0e77406cfb7ff85437ec02b769c6d748e3d9ab5e7773fdd3"
Oct 30 13:29:18.704913 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.704890    5893 generic.go:356] "Generic (PLEG): container finished" podID="2912d932-69b4-407f-91a9-a18cfab1ca79" containerID="ae784a2ed88d0950d94fb13f98b70af83db26e4fb537734e03eb2fbb2faa42a6" exitCode=0
Oct 30 13:29:18.705029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.704942    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"
Oct 30 13:29:18.705085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.705075    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x" event={"ID":"2912d932-69b4-407f-91a9-a18cfab1ca79","Type":"ContainerDied","Data":"ae784a2ed88d0950d94fb13f98b70af83db26e4fb537734e03eb2fbb2faa42a6"}
Oct 30 13:29:18.705143 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.705108    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x" event={"ID":"2912d932-69b4-407f-91a9-a18cfab1ca79","Type":"ContainerDied","Data":"044fe14a82ef07e0a391620e28df886dc3794d2c342a73f15fa47bd4d6693de2"}
Oct 30 13:29:18.714473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.714453    5893 scope.go:122] "RemoveContainer" containerID="839ece2ba1ba9cac0e77406cfb7ff85437ec02b769c6d748e3d9ab5e7773fdd3"
Oct 30 13:29:18.714771 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:29:18.714749    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"839ece2ba1ba9cac0e77406cfb7ff85437ec02b769c6d748e3d9ab5e7773fdd3\": container with ID starting with 839ece2ba1ba9cac0e77406cfb7ff85437ec02b769c6d748e3d9ab5e7773fdd3 not found: ID does not exist" containerID="839ece2ba1ba9cac0e77406cfb7ff85437ec02b769c6d748e3d9ab5e7773fdd3"
Oct 30 13:29:18.714858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.714783    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"839ece2ba1ba9cac0e77406cfb7ff85437ec02b769c6d748e3d9ab5e7773fdd3"} err="failed to get container status \"839ece2ba1ba9cac0e77406cfb7ff85437ec02b769c6d748e3d9ab5e7773fdd3\": rpc error: code = NotFound desc = could not find container \"839ece2ba1ba9cac0e77406cfb7ff85437ec02b769c6d748e3d9ab5e7773fdd3\": container with ID starting with 839ece2ba1ba9cac0e77406cfb7ff85437ec02b769c6d748e3d9ab5e7773fdd3 not found: ID does not exist"
Oct 30 13:29:18.714858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.714808    5893 scope.go:122] "RemoveContainer" containerID="ae784a2ed88d0950d94fb13f98b70af83db26e4fb537734e03eb2fbb2faa42a6"
Oct 30 13:29:18.724318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.724296    5893 scope.go:122] "RemoveContainer" containerID="ae784a2ed88d0950d94fb13f98b70af83db26e4fb537734e03eb2fbb2faa42a6"
Oct 30 13:29:18.724617 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:29:18.724598    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae784a2ed88d0950d94fb13f98b70af83db26e4fb537734e03eb2fbb2faa42a6\": container with ID starting with ae784a2ed88d0950d94fb13f98b70af83db26e4fb537734e03eb2fbb2faa42a6 not found: ID does not exist" containerID="ae784a2ed88d0950d94fb13f98b70af83db26e4fb537734e03eb2fbb2faa42a6"
Oct 30 13:29:18.724694 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.724624    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae784a2ed88d0950d94fb13f98b70af83db26e4fb537734e03eb2fbb2faa42a6"} err="failed to get container status \"ae784a2ed88d0950d94fb13f98b70af83db26e4fb537734e03eb2fbb2faa42a6\": rpc error: code = NotFound desc = could not find container \"ae784a2ed88d0950d94fb13f98b70af83db26e4fb537734e03eb2fbb2faa42a6\": container with ID starting with ae784a2ed88d0950d94fb13f98b70af83db26e4fb537734e03eb2fbb2faa42a6 not found: ID does not exist"
Oct 30 13:29:18.763023 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.762994    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"]
Oct 30 13:29:18.768976 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.768951    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-2afe58/auth-test-model-2afe58-kserve-564986ff9c-nfv9g"]
Oct 30 13:29:18.810870 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.810814    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"]
Oct 30 13:29:18.816210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:18.816185    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-acd125/sub-test-model-acd125-kserve-56bb8887b5-4rn5x"]
Oct 30 13:29:20.532469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:20.532434    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2912d932-69b4-407f-91a9-a18cfab1ca79" path="/var/lib/kubelet/pods/2912d932-69b4-407f-91a9-a18cfab1ca79/volumes"
Oct 30 13:29:20.533115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:20.533094    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da8cba62-e3df-4293-968a-3ce7be6b699a" path="/var/lib/kubelet/pods/da8cba62-e3df-4293-968a-3ce7be6b699a/volumes"
Oct 30 13:29:22.214228 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.214196    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9077-4ce42c"]
Oct 30 13:29:22.214932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.214917    5893 cpu_manager.go:414] "containerMap: removing container" podUID="2912d932-69b4-407f-91a9-a18cfab1ca79" containerName="main"
Oct 30 13:29:22.214977 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.214933    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2912d932-69b4-407f-91a9-a18cfab1ca79" containerName="main"
Oct 30 13:29:22.215059 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.214981    5893 cpu_manager.go:414] "containerMap: removing container" podUID="da8cba62-e3df-4293-968a-3ce7be6b699a" containerName="main"
Oct 30 13:29:22.215059 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.214988    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da8cba62-e3df-4293-968a-3ce7be6b699a" containerName="main"
Oct 30 13:29:22.215138 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.215076    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="da8cba62-e3df-4293-968a-3ce7be6b699a" containerName="main"
Oct 30 13:29:22.215138 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.215090    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="2912d932-69b4-407f-91a9-a18cfab1ca79" containerName="main"
Oct 30 13:29:22.220344 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.220323    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-4ce42c"
Oct 30 13:29:22.238107 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.238079    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9077-4ce42c"]
Oct 30 13:29:22.246390 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.246364    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8xhdg\""
Oct 30 13:29:22.341042 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.341009    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m24c\" (UniqueName: \"kubernetes.io/projected/97168f94-2f97-4543-a1c7-1517adab5968-kube-api-access-5m24c\") pod \"test-curl-9077-4ce42c\" (UID: \"97168f94-2f97-4543-a1c7-1517adab5968\") " pod="opendatahub/test-curl-9077-4ce42c"
Oct 30 13:29:22.442877 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.442844    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m24c\" (UniqueName: \"kubernetes.io/projected/97168f94-2f97-4543-a1c7-1517adab5968-kube-api-access-5m24c\") pod \"test-curl-9077-4ce42c\" (UID: \"97168f94-2f97-4543-a1c7-1517adab5968\") " pod="opendatahub/test-curl-9077-4ce42c"
Oct 30 13:29:22.466366 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.466289    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m24c\" (UniqueName: \"kubernetes.io/projected/97168f94-2f97-4543-a1c7-1517adab5968-kube-api-access-5m24c\") pod \"test-curl-9077-4ce42c\" (UID: \"97168f94-2f97-4543-a1c7-1517adab5968\") " pod="opendatahub/test-curl-9077-4ce42c"
Oct 30 13:29:22.533837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.533809    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-4ce42c"
Oct 30 13:29:22.916145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:22.916119    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9077-4ce42c"]
Oct 30 13:29:22.918357 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:29:22.918310    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97168f94_2f97_4543_a1c7_1517adab5968.slice/crio-b372b6eb4bdd1596202489452773e01b5205086376b6c5d3efe157d002987734 WatchSource:0}: Error finding container b372b6eb4bdd1596202489452773e01b5205086376b6c5d3efe157d002987734: Status 404 returned error can't find the container with id b372b6eb4bdd1596202489452773e01b5205086376b6c5d3efe157d002987734
Oct 30 13:29:23.753317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:23.753255    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9077-4ce42c" event={"ID":"97168f94-2f97-4543-a1c7-1517adab5968","Type":"ContainerStarted","Data":"e27ff1800a62ebedd027f4464dd9ec33c57a55fcfa191002903c82d3cf030ab4"}
Oct 30 13:29:23.753800 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:23.753335    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9077-4ce42c" event={"ID":"97168f94-2f97-4543-a1c7-1517adab5968","Type":"ContainerStarted","Data":"b372b6eb4bdd1596202489452773e01b5205086376b6c5d3efe157d002987734"}
Oct 30 13:29:23.797517 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:23.797462    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9077-4ce42c" podStartSLOduration=1.797445787 podStartE2EDuration="1.797445787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:23.795408037 +0000 UTC m=+2605.378837730" watchObservedRunningTime="2026-10-30 13:29:23.797445787 +0000 UTC m=+2605.380875479"
Oct 30 13:29:24.787653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:24.787621    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9077-4ce42c"]
Oct 30 13:29:24.804433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:24.804402    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9077-4ce42c"]
Oct 30 13:29:24.984172 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:24.984139    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9077-7c55e1"]
Oct 30 13:29:24.984910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:24.984892    5893 cpu_manager.go:414] "containerMap: removing container" podUID="97168f94-2f97-4543-a1c7-1517adab5968" containerName="test-curl-9077-4ce42c"
Oct 30 13:29:24.984985 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:24.984911    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97168f94-2f97-4543-a1c7-1517adab5968" containerName="test-curl-9077-4ce42c"
Oct 30 13:29:24.985036 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:24.985026    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="97168f94-2f97-4543-a1c7-1517adab5968" containerName="test-curl-9077-4ce42c"
Oct 30 13:29:24.988590 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:24.988569    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-7c55e1"
Oct 30 13:29:25.007674 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:25.007652    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9077-7c55e1"]
Oct 30 13:29:25.108115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:25.108030    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5fgd\" (UniqueName: \"kubernetes.io/projected/f0660495-cede-4cb4-b50c-2edc79e7526c-kube-api-access-v5fgd\") pod \"test-curl-9077-7c55e1\" (UID: \"f0660495-cede-4cb4-b50c-2edc79e7526c\") " pod="opendatahub/test-curl-9077-7c55e1"
Oct 30 13:29:25.209641 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:25.209604    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5fgd\" (UniqueName: \"kubernetes.io/projected/f0660495-cede-4cb4-b50c-2edc79e7526c-kube-api-access-v5fgd\") pod \"test-curl-9077-7c55e1\" (UID: \"f0660495-cede-4cb4-b50c-2edc79e7526c\") " pod="opendatahub/test-curl-9077-7c55e1"
Oct 30 13:29:25.235125 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:25.235094    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5fgd\" (UniqueName: \"kubernetes.io/projected/f0660495-cede-4cb4-b50c-2edc79e7526c-kube-api-access-v5fgd\") pod \"test-curl-9077-7c55e1\" (UID: \"f0660495-cede-4cb4-b50c-2edc79e7526c\") " pod="opendatahub/test-curl-9077-7c55e1"
Oct 30 13:29:25.302799 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:25.302770    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-7c55e1"
Oct 30 13:29:25.627490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:25.626120    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9077-7c55e1"]
Oct 30 13:29:25.637671 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:29:25.637629    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0660495_cede_4cb4_b50c_2edc79e7526c.slice/crio-548b19c4c170a801eb963bc731d2dbc921e6619c3842f7d6619765d06175385e WatchSource:0}: Error finding container 548b19c4c170a801eb963bc731d2dbc921e6619c3842f7d6619765d06175385e: Status 404 returned error can't find the container with id 548b19c4c170a801eb963bc731d2dbc921e6619c3842f7d6619765d06175385e
Oct 30 13:29:25.790866 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:25.790820    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9077-7c55e1" event={"ID":"f0660495-cede-4cb4-b50c-2edc79e7526c","Type":"ContainerStarted","Data":"15b83e69bd8003ce7ce54d7163b48d28c682c182f3df79ee4ea57f33c48b3e04"}
Oct 30 13:29:25.791370 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:25.790875    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9077-7c55e1" event={"ID":"f0660495-cede-4cb4-b50c-2edc79e7526c","Type":"ContainerStarted","Data":"548b19c4c170a801eb963bc731d2dbc921e6619c3842f7d6619765d06175385e"}
Oct 30 13:29:25.791370 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:25.790958    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9077-4ce42c" podUID="97168f94-2f97-4543-a1c7-1517adab5968" containerName="test-curl-9077-4ce42c" containerID="cri-o://e27ff1800a62ebedd027f4464dd9ec33c57a55fcfa191002903c82d3cf030ab4" gracePeriod=2
Oct 30 13:29:25.804082 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:29:25.804051    5893 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-4ce42c\" is forbidden: User \"system:node:ip-10-0-141-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-232.ec2.internal' and this object" podUID="97168f94-2f97-4543-a1c7-1517adab5968" pod="opendatahub/test-curl-9077-4ce42c"
Oct 30 13:29:25.841650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:25.841595    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9077-7c55e1" podStartSLOduration=1.841577518 podStartE2EDuration="1.841577518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:25.840131624 +0000 UTC m=+2607.423561317" watchObservedRunningTime="2026-10-30 13:29:25.841577518 +0000 UTC m=+2607.425007211"
Oct 30 13:29:25.853197 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:29:25.853159    5893 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-4ce42c\" is forbidden: User \"system:node:ip-10-0-141-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-232.ec2.internal' and this object" podUID="97168f94-2f97-4543-a1c7-1517adab5968" pod="opendatahub/test-curl-9077-4ce42c"
Oct 30 13:29:26.522299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:26.521433    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"]
Oct 30 13:29:26.612297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:26.607048    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"
Oct 30 13:29:26.632622 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:29:26.632564    5893 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-4ce42c\" is forbidden: User \"system:node:ip-10-0-141-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-232.ec2.internal' and this object" podUID="97168f94-2f97-4543-a1c7-1517adab5968" pod="opendatahub/test-curl-9077-4ce42c"
Oct 30 13:29:26.633402 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:26.632629    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-76405c-dockercfg-gvtnv\""
Oct 30 13:29:26.637623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:26.636928    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qt9p\" (UniqueName: \"kubernetes.io/projected/e65e23b4-252a-4316-8b34-87fec8e13f8b-kube-api-access-2qt9p\") pod \"payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb\" (UID: \"e65e23b4-252a-4316-8b34-87fec8e13f8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"
Oct 30 13:29:26.637623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:26.636977    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65e23b4-252a-4316-8b34-87fec8e13f8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb\" (UID: \"e65e23b4-252a-4316-8b34-87fec8e13f8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"
Oct 30 13:29:26.638514 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:26.638462    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"]
Oct 30 13:29:26.644391 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:26.644357    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-76405c\""
Oct 30 13:29:26.738525 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:26.738493    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qt9p\" (UniqueName: \"kubernetes.io/projected/e65e23b4-252a-4316-8b34-87fec8e13f8b-kube-api-access-2qt9p\") pod \"payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb\" (UID: \"e65e23b4-252a-4316-8b34-87fec8e13f8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"
Oct 30 13:29:26.738706 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:26.738536    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65e23b4-252a-4316-8b34-87fec8e13f8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb\" (UID: \"e65e23b4-252a-4316-8b34-87fec8e13f8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"
Oct 30 13:29:26.739487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:26.739467    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65e23b4-252a-4316-8b34-87fec8e13f8b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb\" (UID: \"e65e23b4-252a-4316-8b34-87fec8e13f8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"
Oct 30 13:29:26.761784 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:26.761749    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qt9p\" (UniqueName: \"kubernetes.io/projected/e65e23b4-252a-4316-8b34-87fec8e13f8b-kube-api-access-2qt9p\") pod \"payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb\" (UID: \"e65e23b4-252a-4316-8b34-87fec8e13f8b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"
Oct 30 13:29:26.901766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:26.901683    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9077-7c55e1"]
Oct 30 13:29:26.912552 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:26.912518    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9077-7c55e1"]
Oct 30 13:29:26.956966 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:26.956934    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"
Oct 30 13:29:27.271243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:27.270989    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"]
Oct 30 13:29:27.274000 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:29:27.273974    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65e23b4_252a_4316_8b34_87fec8e13f8b.slice/crio-ad9441be97350104c06c0d343c882f11167d4c1e9e0c6aed299d6cefd748c86a WatchSource:0}: Error finding container ad9441be97350104c06c0d343c882f11167d4c1e9e0c6aed299d6cefd748c86a: Status 404 returned error can't find the container with id ad9441be97350104c06c0d343c882f11167d4c1e9e0c6aed299d6cefd748c86a
Oct 30 13:29:27.809516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:27.809482    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb" event={"ID":"e65e23b4-252a-4316-8b34-87fec8e13f8b","Type":"ContainerStarted","Data":"46292febe9458a4f44965e49ef403da9efd077cc01cef92a3252a96c485239f9"}
Oct 30 13:29:27.809881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:27.809849    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb" event={"ID":"e65e23b4-252a-4316-8b34-87fec8e13f8b","Type":"ContainerStarted","Data":"ad9441be97350104c06c0d343c882f11167d4c1e9e0c6aed299d6cefd748c86a"}
Oct 30 13:29:27.810013 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:27.809899    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"
Oct 30 13:29:27.810013 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:27.809773    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9077-7c55e1" podUID="f0660495-cede-4cb4-b50c-2edc79e7526c" containerName="test-curl-9077-7c55e1" containerID="cri-o://15b83e69bd8003ce7ce54d7163b48d28c682c182f3df79ee4ea57f33c48b3e04" gracePeriod=2
Oct 30 13:29:27.853535 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:27.853487    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb" podStartSLOduration=1.853473947 podStartE2EDuration="1.853473947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:27.852189664 +0000 UTC m=+2609.435619377" watchObservedRunningTime="2026-10-30 13:29:27.853473947 +0000 UTC m=+2609.436903639"
Oct 30 13:29:27.866531 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:29:27.866449    5893 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-7c55e1\" is forbidden: User \"system:node:ip-10-0-141-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-232.ec2.internal' and this object" podUID="f0660495-cede-4cb4-b50c-2edc79e7526c" pod="opendatahub/test-curl-9077-7c55e1"
Oct 30 13:29:28.000611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:28.000586    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-4ce42c"
Oct 30 13:29:28.014505 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:29:28.014467    5893 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-7c55e1\" is forbidden: User \"system:node:ip-10-0-141-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-232.ec2.internal' and this object" podUID="f0660495-cede-4cb4-b50c-2edc79e7526c" pod="opendatahub/test-curl-9077-7c55e1"
Oct 30 13:29:28.060219 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:28.060125    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97168f94-2f97-4543-a1c7-1517adab5968-kube-api-access-5m24c\" (UniqueName: \"kubernetes.io/projected/97168f94-2f97-4543-a1c7-1517adab5968-kube-api-access-5m24c\") pod \"97168f94-2f97-4543-a1c7-1517adab5968\" (UID: \"97168f94-2f97-4543-a1c7-1517adab5968\") "
Oct 30 13:29:28.062511 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:28.062480    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97168f94-2f97-4543-a1c7-1517adab5968-kube-api-access-5m24c" pod "97168f94-2f97-4543-a1c7-1517adab5968" (UID: "97168f94-2f97-4543-a1c7-1517adab5968"). InnerVolumeSpecName "kube-api-access-5m24c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:28.162436 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:28.162403    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m24c\" (UniqueName: \"kubernetes.io/projected/97168f94-2f97-4543-a1c7-1517adab5968-kube-api-access-5m24c\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:29:28.532436 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:28.532401    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97168f94-2f97-4543-a1c7-1517adab5968" path="/var/lib/kubelet/pods/97168f94-2f97-4543-a1c7-1517adab5968/volumes"
Oct 30 13:29:28.540612 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:29:28.540581    5893 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9077-7c55e1\" is forbidden: User \"system:node:ip-10-0-141-232.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-232.ec2.internal' and this object" podUID="f0660495-cede-4cb4-b50c-2edc79e7526c" pod="opendatahub/test-curl-9077-7c55e1"
Oct 30 13:29:28.818622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:28.818530    5893 generic.go:356] "Generic (PLEG): container finished" podID="97168f94-2f97-4543-a1c7-1517adab5968" containerID="e27ff1800a62ebedd027f4464dd9ec33c57a55fcfa191002903c82d3cf030ab4" exitCode=137
Oct 30 13:29:28.818622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:28.818572    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-4ce42c"
Oct 30 13:29:28.818876 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:28.818626    5893 scope.go:122] "RemoveContainer" containerID="e27ff1800a62ebedd027f4464dd9ec33c57a55fcfa191002903c82d3cf030ab4"
Oct 30 13:29:28.829357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:28.829331    5893 scope.go:122] "RemoveContainer" containerID="e27ff1800a62ebedd027f4464dd9ec33c57a55fcfa191002903c82d3cf030ab4"
Oct 30 13:29:28.829665 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:29:28.829645    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e27ff1800a62ebedd027f4464dd9ec33c57a55fcfa191002903c82d3cf030ab4\": container with ID starting with e27ff1800a62ebedd027f4464dd9ec33c57a55fcfa191002903c82d3cf030ab4 not found: ID does not exist" containerID="e27ff1800a62ebedd027f4464dd9ec33c57a55fcfa191002903c82d3cf030ab4"
Oct 30 13:29:28.829758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:28.829677    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e27ff1800a62ebedd027f4464dd9ec33c57a55fcfa191002903c82d3cf030ab4"} err="failed to get container status \"e27ff1800a62ebedd027f4464dd9ec33c57a55fcfa191002903c82d3cf030ab4\": rpc error: code = NotFound desc = could not find container \"e27ff1800a62ebedd027f4464dd9ec33c57a55fcfa191002903c82d3cf030ab4\": container with ID starting with e27ff1800a62ebedd027f4464dd9ec33c57a55fcfa191002903c82d3cf030ab4 not found: ID does not exist"
Oct 30 13:29:30.049600 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:30.049564    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-7c55e1"
Oct 30 13:29:30.109707 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:30.109616    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0660495-cede-4cb4-b50c-2edc79e7526c-kube-api-access-v5fgd\" (UniqueName: \"kubernetes.io/projected/f0660495-cede-4cb4-b50c-2edc79e7526c-kube-api-access-v5fgd\") pod \"f0660495-cede-4cb4-b50c-2edc79e7526c\" (UID: \"f0660495-cede-4cb4-b50c-2edc79e7526c\") "
Oct 30 13:29:30.112346 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:30.112306    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0660495-cede-4cb4-b50c-2edc79e7526c-kube-api-access-v5fgd" pod "f0660495-cede-4cb4-b50c-2edc79e7526c" (UID: "f0660495-cede-4cb4-b50c-2edc79e7526c"). InnerVolumeSpecName "kube-api-access-v5fgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:30.212282 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:30.212245    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5fgd\" (UniqueName: \"kubernetes.io/projected/f0660495-cede-4cb4-b50c-2edc79e7526c-kube-api-access-v5fgd\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:29:30.534196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:30.534160    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0660495-cede-4cb4-b50c-2edc79e7526c" path="/var/lib/kubelet/pods/f0660495-cede-4cb4-b50c-2edc79e7526c/volumes"
Oct 30 13:29:30.836883 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:30.836778    5893 generic.go:356] "Generic (PLEG): container finished" podID="f0660495-cede-4cb4-b50c-2edc79e7526c" containerID="15b83e69bd8003ce7ce54d7163b48d28c682c182f3df79ee4ea57f33c48b3e04" exitCode=137
Oct 30 13:29:30.836883 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:30.836822    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9077-7c55e1"
Oct 30 13:29:30.837198 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:30.836884    5893 scope.go:122] "RemoveContainer" containerID="15b83e69bd8003ce7ce54d7163b48d28c682c182f3df79ee4ea57f33c48b3e04"
Oct 30 13:29:30.857167 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:30.857139    5893 scope.go:122] "RemoveContainer" containerID="15b83e69bd8003ce7ce54d7163b48d28c682c182f3df79ee4ea57f33c48b3e04"
Oct 30 13:29:30.857699 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:29:30.857661    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b83e69bd8003ce7ce54d7163b48d28c682c182f3df79ee4ea57f33c48b3e04\": container with ID starting with 15b83e69bd8003ce7ce54d7163b48d28c682c182f3df79ee4ea57f33c48b3e04 not found: ID does not exist" containerID="15b83e69bd8003ce7ce54d7163b48d28c682c182f3df79ee4ea57f33c48b3e04"
Oct 30 13:29:30.857825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:30.857709    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b83e69bd8003ce7ce54d7163b48d28c682c182f3df79ee4ea57f33c48b3e04"} err="failed to get container status \"15b83e69bd8003ce7ce54d7163b48d28c682c182f3df79ee4ea57f33c48b3e04\": rpc error: code = NotFound desc = could not find container \"15b83e69bd8003ce7ce54d7163b48d28c682c182f3df79ee4ea57f33c48b3e04\": container with ID starting with 15b83e69bd8003ce7ce54d7163b48d28c682c182f3df79ee4ea57f33c48b3e04 not found: ID does not exist"
Oct 30 13:29:34.397100 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.397061    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"]
Oct 30 13:29:34.397933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.397913    5893 cpu_manager.go:414] "containerMap: removing container" podUID="f0660495-cede-4cb4-b50c-2edc79e7526c" containerName="test-curl-9077-7c55e1"
Oct 30 13:29:34.397933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.397933    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0660495-cede-4cb4-b50c-2edc79e7526c" containerName="test-curl-9077-7c55e1"
Oct 30 13:29:34.398129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.398118    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0660495-cede-4cb4-b50c-2edc79e7526c" containerName="test-curl-9077-7c55e1"
Oct 30 13:29:34.403934 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.403914    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.433726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.433691    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"]
Oct 30 13:29:34.441326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.441042    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-96ab73\"/\"openshift-service-ca.crt\""
Oct 30 13:29:34.442558 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.441781    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-96ab73\"/\"default-dockercfg-mc7p9\""
Oct 30 13:29:34.442558 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.441905    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-96ab73\"/\"auth-test-model-96ab73-kserve-self-signed-certs\""
Oct 30 13:29:34.442558 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.442001    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-96ab73\"/\"kube-root-ca.crt\""
Oct 30 13:29:34.486176 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.486143    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-dshm\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.486374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.486214    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-model-cache\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.486374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.486306    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-tmp-dir\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.486374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.486348    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-home\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.486561 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.486425    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggw6c\" (UniqueName: \"kubernetes.io/projected/7d032dc0-e641-447b-bce2-c46b5968cb6f-kube-api-access-ggw6c\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.486561 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.486509    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d032dc0-e641-447b-bce2-c46b5968cb6f-tls-certs\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.587734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.587701    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-dshm\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.587921 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.587868    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-model-cache\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.588001 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.587947    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-tmp-dir\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.588065 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.588026    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-home\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.588121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.588074    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggw6c\" (UniqueName: \"kubernetes.io/projected/7d032dc0-e641-447b-bce2-c46b5968cb6f-kube-api-access-ggw6c\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.588532 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.588179    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d032dc0-e641-447b-bce2-c46b5968cb6f-tls-certs\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.588532 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.588226    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-model-cache\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.588532 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.588304    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-tmp-dir\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.588532 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.588468    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-home\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.590245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.590218    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-dshm\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.590985 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.590966    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d032dc0-e641-447b-bce2-c46b5968cb6f-tls-certs\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.638904 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.638868    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggw6c\" (UniqueName: \"kubernetes.io/projected/7d032dc0-e641-447b-bce2-c46b5968cb6f-kube-api-access-ggw6c\") pod \"auth-test-model-96ab73-kserve-5b689558d5-2rdbj\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") " pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:34.744078 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:34.744043    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:35.132637 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:35.132599    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"]
Oct 30 13:29:35.134523 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:29:35.134493    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d032dc0_e641_447b_bce2_c46b5968cb6f.slice/crio-025395ca47af642c0e3fa7c7a17a9bb381be21395b75dced87001a25db4d932e WatchSource:0}: Error finding container 025395ca47af642c0e3fa7c7a17a9bb381be21395b75dced87001a25db4d932e: Status 404 returned error can't find the container with id 025395ca47af642c0e3fa7c7a17a9bb381be21395b75dced87001a25db4d932e
Oct 30 13:29:35.879861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:35.879824    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj" event={"ID":"7d032dc0-e641-447b-bce2-c46b5968cb6f","Type":"ContainerStarted","Data":"8994f55d7adab87dbf363d2e45298b08091fa4db8bea56395ec1574a02482d71"}
Oct 30 13:29:35.879861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:35.879877    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj" event={"ID":"7d032dc0-e641-447b-bce2-c46b5968cb6f","Type":"ContainerStarted","Data":"025395ca47af642c0e3fa7c7a17a9bb381be21395b75dced87001a25db4d932e"}
Oct 30 13:29:35.922808 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:35.922760    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj" podStartSLOduration=1.922746145 podStartE2EDuration="1.922746145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:35.920288822 +0000 UTC m=+2617.503718516" watchObservedRunningTime="2026-10-30 13:29:35.922746145 +0000 UTC m=+2617.506175888"
Oct 30 13:29:36.744243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:36.744216    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:38.635505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.635424    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"]
Oct 30 13:29:38.639888 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.639868    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.661433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.661410    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-0e874e-openshift-default-dockercfg-4czbc\""
Oct 30 13:29:38.676527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.676496    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"]
Oct 30 13:29:38.770885 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.770847    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-envoy\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.771111 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.770898    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-data\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.771111 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.771033    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-credential-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.771111 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.771083    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-workload-certs\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.771343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.771210    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9vmh\" (UniqueName: \"kubernetes.io/projected/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-kube-api-access-l9vmh\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.771388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.771363    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-token\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.771435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.771421    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istiod-ca-cert\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.771521 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.771506    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-podinfo\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.771600 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.771589    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-workload-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.822489 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.822458    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"
Oct 30 13:29:38.873438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.873260    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istiod-ca-cert\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.874556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.874524    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istiod-ca-cert\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.874727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.874609    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-podinfo\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.874727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.874704    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-workload-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.874853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.874779    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-envoy\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.874853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.874840    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-data\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.874962 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.874880    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-credential-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.874962 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.874906    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-workload-certs\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.874962 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.874952    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9vmh\" (UniqueName: \"kubernetes.io/projected/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-kube-api-access-l9vmh\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.875140 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.875017    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-token\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.876317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.876249    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-credential-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.876482 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.876386    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-data\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.876582 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.876556    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-workload-certs\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.877187 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.877159    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-workload-socket\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.878643 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.878617    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-podinfo\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.879239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.879214    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-envoy\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.907263 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.907151    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9vmh\" (UniqueName: \"kubernetes.io/projected/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-kube-api-access-l9vmh\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.917170 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.917133    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-token\") pod \"e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") " pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:38.955383 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:38.955356    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:39.300765 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:39.300734    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"]
Oct 30 13:29:39.300964 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:29:39.300931    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf79c5dcd_ddd1_44c2_9f5e_93b8e08cc81d.slice/crio-db8c62f48d0612875e5b103ecf3e7a411a8ace6b01a5cef13cab996d0e3f9807 WatchSource:0}: Error finding container db8c62f48d0612875e5b103ecf3e7a411a8ace6b01a5cef13cab996d0e3f9807: Status 404 returned error can't find the container with id db8c62f48d0612875e5b103ecf3e7a411a8ace6b01a5cef13cab996d0e3f9807
Oct 30 13:29:39.306732 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:39.306686    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:29:39.306858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:39.306836    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:29:39.306951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:39.306927    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:29:39.930911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:39.924222    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws" event={"ID":"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d","Type":"ContainerStarted","Data":"5da8786df647cbdbd6f953bc7ac7d16aa57775703b07ada147a2dabeb0c94961"}
Oct 30 13:29:39.930911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:39.924311    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws" event={"ID":"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d","Type":"ContainerStarted","Data":"db8c62f48d0612875e5b103ecf3e7a411a8ace6b01a5cef13cab996d0e3f9807"}
Oct 30 13:29:39.964293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:39.963962    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:40.971300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:40.969421    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:29:40.971300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:40.969481    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws" podUID="f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 13:29:41.966442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:41.966406    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:29:41.966718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:41.966693    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws" podUID="f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 13:29:42.959300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:42.956265    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:29:42.959300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:42.956346    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws" podUID="f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 13:29:43.957299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:43.956702    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:29:43.957299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:43.956784    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws" podUID="f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 13:29:44.754297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:44.753118    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:44.783537 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:44.783506    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:44.827004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:44.826953    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws" podStartSLOduration=6.82693814 podStartE2EDuration="6.82693814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:39.999849596 +0000 UTC m=+2621.583279290" watchObservedRunningTime="2026-10-30 13:29:44.82693814 +0000 UTC m=+2626.410367834"
Oct 30 13:29:44.959300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:44.958478    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:29:44.959300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:44.958549    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws" podUID="f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.170:15021/healthz/ready\": dial tcp 10.134.0.170:15021: connect: connection refused"
Oct 30 13:29:45.003228 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:45.003203    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:29:46.232129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:46.232097    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:46.232961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:46.232871    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:46.235160 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:29:46.235135    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:29:48.646022 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:29:48.645978    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:30:00.491304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.450483    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"]
Oct 30 13:30:00.628875 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.628754    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"
Oct 30 13:30:00.690048 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.684977    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:30:00.701552 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.701514    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bmps\" (UniqueName: \"kubernetes.io/projected/78c9276f-5e11-4866-8a30-1bb4c38a5e8a-kube-api-access-5bmps\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd\" (UID: \"78c9276f-5e11-4866-8a30-1bb4c38a5e8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"
Oct 30 13:30:00.701887 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.701865    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9276f-5e11-4866-8a30-1bb4c38a5e8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd\" (UID: \"78c9276f-5e11-4866-8a30-1bb4c38a5e8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"
Oct 30 13:30:00.714437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.714412    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"]
Oct 30 13:30:00.730408 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.730380    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"]
Oct 30 13:30:00.754054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.753967    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"
Oct 30 13:30:00.787296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.770343    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"]
Oct 30 13:30:00.787296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.770475    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"
Oct 30 13:30:00.832296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.824190    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppdmd\" (UniqueName: \"kubernetes.io/projected/871e8c5b-99ee-48f1-8a0a-ae1846253ec2-kube-api-access-ppdmd\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2\" (UID: \"871e8c5b-99ee-48f1-8a0a-ae1846253ec2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"
Oct 30 13:30:00.832296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.824242    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8swbp\" (UniqueName: \"kubernetes.io/projected/c9484142-6e89-4dc7-89ff-0fcd438cfedb-kube-api-access-8swbp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk\" (UID: \"c9484142-6e89-4dc7-89ff-0fcd438cfedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"
Oct 30 13:30:00.832296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.824312    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9484142-6e89-4dc7-89ff-0fcd438cfedb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk\" (UID: \"c9484142-6e89-4dc7-89ff-0fcd438cfedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"
Oct 30 13:30:00.832296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.824344    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bmps\" (UniqueName: \"kubernetes.io/projected/78c9276f-5e11-4866-8a30-1bb4c38a5e8a-kube-api-access-5bmps\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd\" (UID: \"78c9276f-5e11-4866-8a30-1bb4c38a5e8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"
Oct 30 13:30:00.832296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.824430    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9276f-5e11-4866-8a30-1bb4c38a5e8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd\" (UID: \"78c9276f-5e11-4866-8a30-1bb4c38a5e8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"
Oct 30 13:30:00.832296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.824456    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871e8c5b-99ee-48f1-8a0a-ae1846253ec2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2\" (UID: \"871e8c5b-99ee-48f1-8a0a-ae1846253ec2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"
Oct 30 13:30:00.832296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.825721    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9276f-5e11-4866-8a30-1bb4c38a5e8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd\" (UID: \"78c9276f-5e11-4866-8a30-1bb4c38a5e8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"
Oct 30 13:30:00.838963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.838935    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"]
Oct 30 13:30:00.844672 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.844645    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"]
Oct 30 13:30:00.861442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.860440    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"]
Oct 30 13:30:00.861442 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.860561    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"
Oct 30 13:30:00.879811 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.878239    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"]
Oct 30 13:30:00.935011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.933639    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppdmd\" (UniqueName: \"kubernetes.io/projected/871e8c5b-99ee-48f1-8a0a-ae1846253ec2-kube-api-access-ppdmd\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2\" (UID: \"871e8c5b-99ee-48f1-8a0a-ae1846253ec2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"
Oct 30 13:30:00.935011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.933688    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8swbp\" (UniqueName: \"kubernetes.io/projected/c9484142-6e89-4dc7-89ff-0fcd438cfedb-kube-api-access-8swbp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk\" (UID: \"c9484142-6e89-4dc7-89ff-0fcd438cfedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"
Oct 30 13:30:00.935011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.933743    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnw8m\" (UniqueName: \"kubernetes.io/projected/8d60b5df-0998-4618-b201-4793314d793d-kube-api-access-jnw8m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r\" (UID: \"8d60b5df-0998-4618-b201-4793314d793d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"
Oct 30 13:30:00.935011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.933777    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9484142-6e89-4dc7-89ff-0fcd438cfedb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk\" (UID: \"c9484142-6e89-4dc7-89ff-0fcd438cfedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"
Oct 30 13:30:00.935011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.933869    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871e8c5b-99ee-48f1-8a0a-ae1846253ec2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2\" (UID: \"871e8c5b-99ee-48f1-8a0a-ae1846253ec2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"
Oct 30 13:30:00.935011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.933987    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d60b5df-0998-4618-b201-4793314d793d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r\" (UID: \"8d60b5df-0998-4618-b201-4793314d793d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"
Oct 30 13:30:00.936967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.936916    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871e8c5b-99ee-48f1-8a0a-ae1846253ec2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2\" (UID: \"871e8c5b-99ee-48f1-8a0a-ae1846253ec2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"
Oct 30 13:30:00.940832 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.940714    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bmps\" (UniqueName: \"kubernetes.io/projected/78c9276f-5e11-4866-8a30-1bb4c38a5e8a-kube-api-access-5bmps\") pod \"maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd\" (UID: \"78c9276f-5e11-4866-8a30-1bb4c38a5e8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"
Oct 30 13:30:00.941378 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.941335    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9484142-6e89-4dc7-89ff-0fcd438cfedb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk\" (UID: \"c9484142-6e89-4dc7-89ff-0fcd438cfedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"
Oct 30 13:30:00.994024 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:00.993989    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8swbp\" (UniqueName: \"kubernetes.io/projected/c9484142-6e89-4dc7-89ff-0fcd438cfedb-kube-api-access-8swbp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk\" (UID: \"c9484142-6e89-4dc7-89ff-0fcd438cfedb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"
Oct 30 13:30:01.022742 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:01.016432    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppdmd\" (UniqueName: \"kubernetes.io/projected/871e8c5b-99ee-48f1-8a0a-ae1846253ec2-kube-api-access-ppdmd\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2\" (UID: \"871e8c5b-99ee-48f1-8a0a-ae1846253ec2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"
Oct 30 13:30:01.022742 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:01.021501    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"
Oct 30 13:30:01.041640 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:01.041610    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnw8m\" (UniqueName: \"kubernetes.io/projected/8d60b5df-0998-4618-b201-4793314d793d-kube-api-access-jnw8m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r\" (UID: \"8d60b5df-0998-4618-b201-4793314d793d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"
Oct 30 13:30:01.043161 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:01.043133    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d60b5df-0998-4618-b201-4793314d793d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r\" (UID: \"8d60b5df-0998-4618-b201-4793314d793d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"
Oct 30 13:30:01.046084 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:01.043912    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d60b5df-0998-4618-b201-4793314d793d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r\" (UID: \"8d60b5df-0998-4618-b201-4793314d793d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"
Oct 30 13:30:01.083317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:01.079785    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnw8m\" (UniqueName: \"kubernetes.io/projected/8d60b5df-0998-4618-b201-4793314d793d-kube-api-access-jnw8m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r\" (UID: \"8d60b5df-0998-4618-b201-4793314d793d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"
Oct 30 13:30:01.184833 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:01.184543    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"
Oct 30 13:30:01.187949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:01.185232    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"
Oct 30 13:30:01.235387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:01.233060    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"
Oct 30 13:30:01.598889 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:30:01.598845    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c9276f_5e11_4866_8a30_1bb4c38a5e8a.slice/crio-77623209bb763b84ddd27e45320e97ce3d5090833d36b95d5fbd9d9dd46fbe03 WatchSource:0}: Error finding container 77623209bb763b84ddd27e45320e97ce3d5090833d36b95d5fbd9d9dd46fbe03: Status 404 returned error can't find the container with id 77623209bb763b84ddd27e45320e97ce3d5090833d36b95d5fbd9d9dd46fbe03
Oct 30 13:30:01.605727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:01.605488    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"]
Oct 30 13:30:01.894499 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:01.894302    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"]
Oct 30 13:30:01.913769 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:01.910207    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"]
Oct 30 13:30:01.922152 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:30:01.922115    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d60b5df_0998_4618_b201_4793314d793d.slice/crio-3305f6a54cd9af2215a8adab575b76a5d76a845b41a609dcc272f973370628c3 WatchSource:0}: Error finding container 3305f6a54cd9af2215a8adab575b76a5d76a845b41a609dcc272f973370628c3: Status 404 returned error can't find the container with id 3305f6a54cd9af2215a8adab575b76a5d76a845b41a609dcc272f973370628c3
Oct 30 13:30:02.029830 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:02.029806    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"]
Oct 30 13:30:02.031804 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:30:02.031771    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9484142_6e89_4dc7_89ff_0fcd438cfedb.slice/crio-ec2a77c13a44f83561ca4e5c82ce101b5ec442e9e429fb1c16302ee8ee23032b WatchSource:0}: Error finding container ec2a77c13a44f83561ca4e5c82ce101b5ec442e9e429fb1c16302ee8ee23032b: Status 404 returned error can't find the container with id ec2a77c13a44f83561ca4e5c82ce101b5ec442e9e429fb1c16302ee8ee23032b
Oct 30 13:30:02.279194 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:02.279150    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r" event={"ID":"8d60b5df-0998-4618-b201-4793314d793d","Type":"ContainerStarted","Data":"3305f6a54cd9af2215a8adab575b76a5d76a845b41a609dcc272f973370628c3"}
Oct 30 13:30:02.280797 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:02.280765    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2" event={"ID":"871e8c5b-99ee-48f1-8a0a-ae1846253ec2","Type":"ContainerStarted","Data":"eea49000e00fda0fa5f91df55e72ccb68fd441b43f7e63b4cc10fedac5c843b6"}
Oct 30 13:30:02.282662 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:02.282634    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd" event={"ID":"78c9276f-5e11-4866-8a30-1bb4c38a5e8a","Type":"ContainerStarted","Data":"cc526fdb708f8ab719010b7d57245074e8d548855942c2c4b69f4c6d47ba1918"}
Oct 30 13:30:02.282788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:02.282686    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd" event={"ID":"78c9276f-5e11-4866-8a30-1bb4c38a5e8a","Type":"ContainerStarted","Data":"77623209bb763b84ddd27e45320e97ce3d5090833d36b95d5fbd9d9dd46fbe03"}
Oct 30 13:30:02.284460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:02.284424    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk" event={"ID":"c9484142-6e89-4dc7-89ff-0fcd438cfedb","Type":"ContainerStarted","Data":"ec2a77c13a44f83561ca4e5c82ce101b5ec442e9e429fb1c16302ee8ee23032b"}
Oct 30 13:30:02.327401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:02.327350    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd" podStartSLOduration=2.327334672 podStartE2EDuration="2.327334672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.324224957 +0000 UTC m=+2643.907654649" watchObservedRunningTime="2026-10-30 13:30:02.327334672 +0000 UTC m=+2643.910764415"
Oct 30 13:30:03.296772 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:03.296728    5893 generic.go:356] "Generic (PLEG): container finished" podID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2" containerID="e9d4440c837161a38fe4a650b226bbdd30e32765458c401dbd382a841b694979" exitCode=60
Oct 30 13:30:03.297264 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:03.296804    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2" event={"ID":"871e8c5b-99ee-48f1-8a0a-ae1846253ec2","Type":"ContainerDied","Data":"e9d4440c837161a38fe4a650b226bbdd30e32765458c401dbd382a841b694979"}
Oct 30 13:30:03.297637 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:03.297618    5893 scope.go:122] "RemoveContainer" containerID="e9d4440c837161a38fe4a650b226bbdd30e32765458c401dbd382a841b694979"
Oct 30 13:30:03.298904 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:03.298881    5893 generic.go:356] "Generic (PLEG): container finished" podID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a" containerID="cc526fdb708f8ab719010b7d57245074e8d548855942c2c4b69f4c6d47ba1918" exitCode=60
Oct 30 13:30:03.299075 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:03.299056    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd" event={"ID":"78c9276f-5e11-4866-8a30-1bb4c38a5e8a","Type":"ContainerDied","Data":"cc526fdb708f8ab719010b7d57245074e8d548855942c2c4b69f4c6d47ba1918"}
Oct 30 13:30:03.299352 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:03.299327    5893 scope.go:122] "RemoveContainer" containerID="cc526fdb708f8ab719010b7d57245074e8d548855942c2c4b69f4c6d47ba1918"
Oct 30 13:30:03.302200 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:03.301588    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk" event={"ID":"c9484142-6e89-4dc7-89ff-0fcd438cfedb","Type":"ContainerStarted","Data":"df1ee42222cf0c1e12af316aee8c8b1d9277420eca3ea3e35329af7842708a65"}
Oct 30 13:30:03.302200 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:03.301828    5893 scope.go:122] "RemoveContainer" containerID="df1ee42222cf0c1e12af316aee8c8b1d9277420eca3ea3e35329af7842708a65"
Oct 30 13:30:03.303337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:03.303315    5893 generic.go:356] "Generic (PLEG): container finished" podID="8d60b5df-0998-4618-b201-4793314d793d" containerID="93052a7e508eadd5c2332c8b56759ccc1c56b2919b0080de9b57c6902f8e9055" exitCode=60
Oct 30 13:30:03.303419 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:03.303361    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r" event={"ID":"8d60b5df-0998-4618-b201-4793314d793d","Type":"ContainerDied","Data":"93052a7e508eadd5c2332c8b56759ccc1c56b2919b0080de9b57c6902f8e9055"}
Oct 30 13:30:03.303588 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:03.303576    5893 scope.go:122] "RemoveContainer" containerID="93052a7e508eadd5c2332c8b56759ccc1c56b2919b0080de9b57c6902f8e9055"
Oct 30 13:30:03.985904 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:03.985844    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"]
Oct 30 13:30:04.000363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:03.998509    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:04.037834 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.037807    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-76405c-serving-cert\""
Oct 30 13:30:04.077321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.065384    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400e5f9c-8609-4854-af56-2f31915c6f13-metrics-certs\") pod \"maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:04.077321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.065663    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400e5f9c-8609-4854-af56-2f31915c6f13-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:04.077321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.065729    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4pnf\" (UniqueName: \"kubernetes.io/projected/400e5f9c-8609-4854-af56-2f31915c6f13-kube-api-access-z4pnf\") pod \"maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:04.077321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.065773    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400e5f9c-8609-4854-af56-2f31915c6f13-maas-api-tls\") pod \"maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:04.081322 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.081235    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"]
Oct 30 13:30:04.174311 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.168543    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400e5f9c-8609-4854-af56-2f31915c6f13-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:04.174311 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.168629    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4pnf\" (UniqueName: \"kubernetes.io/projected/400e5f9c-8609-4854-af56-2f31915c6f13-kube-api-access-z4pnf\") pod \"maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:04.174311 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.168674    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400e5f9c-8609-4854-af56-2f31915c6f13-maas-api-tls\") pod \"maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:04.174311 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.168788    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400e5f9c-8609-4854-af56-2f31915c6f13-metrics-certs\") pod \"maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:04.175396 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.175346    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400e5f9c-8609-4854-af56-2f31915c6f13-metrics-certs\") pod \"maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:04.176519 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.176490    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400e5f9c-8609-4854-af56-2f31915c6f13-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:04.180003 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.179986    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400e5f9c-8609-4854-af56-2f31915c6f13-maas-api-tls\") pod \"maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:04.220835 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.220798    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4pnf\" (UniqueName: \"kubernetes.io/projected/400e5f9c-8609-4854-af56-2f31915c6f13-kube-api-access-z4pnf\") pod \"maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:04.316458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.315804    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2" event={"ID":"871e8c5b-99ee-48f1-8a0a-ae1846253ec2","Type":"ContainerStarted","Data":"33f43ea2de0a20cec0498b91cc19017b9ccf8554243f97bfd4e98170189f2c71"}
Oct 30 13:30:04.317997 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.317955    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd" event={"ID":"78c9276f-5e11-4866-8a30-1bb4c38a5e8a","Type":"ContainerStarted","Data":"9cf9e815ee1a749adf7779b2eac7ff002ff0905088cb55e5f86cef6154f0f9e8"}
Oct 30 13:30:04.319937 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.319916    5893 generic.go:356] "Generic (PLEG): container finished" podID="c9484142-6e89-4dc7-89ff-0fcd438cfedb" containerID="df1ee42222cf0c1e12af316aee8c8b1d9277420eca3ea3e35329af7842708a65" exitCode=60
Oct 30 13:30:04.320085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.320002    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk" event={"ID":"c9484142-6e89-4dc7-89ff-0fcd438cfedb","Type":"ContainerDied","Data":"df1ee42222cf0c1e12af316aee8c8b1d9277420eca3ea3e35329af7842708a65"}
Oct 30 13:30:04.320085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.320056    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk" event={"ID":"c9484142-6e89-4dc7-89ff-0fcd438cfedb","Type":"ContainerStarted","Data":"327d5a027a9a190aea0c41daf46d9bee83ee4650eb7fefe1eb5f1555adf6c47d"}
Oct 30 13:30:04.321566 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.321543    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r" event={"ID":"8d60b5df-0998-4618-b201-4793314d793d","Type":"ContainerStarted","Data":"3c9adad7e5ad0f04471a62f656fa0e10823799b4c7578367fa7011ef2de76feb"}
Oct 30 13:30:04.379128 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.378921    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:04.379334 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.379251    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2" podStartSLOduration=4.379233881 podStartE2EDuration="4.379233881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.376805499 +0000 UTC m=+2645.960235216" watchObservedRunningTime="2026-10-30 13:30:04.379233881 +0000 UTC m=+2645.962663575"
Oct 30 13:30:04.434307 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.433143    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r" podStartSLOduration=4.433129934 podStartE2EDuration="4.433129934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.43103655 +0000 UTC m=+2646.014466240" watchObservedRunningTime="2026-10-30 13:30:04.433129934 +0000 UTC m=+2646.016559627"
Oct 30 13:30:04.582438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.582308    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk" podStartSLOduration=4.582286808 podStartE2EDuration="4.582286808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:04.556507755 +0000 UTC m=+2646.139937468" watchObservedRunningTime="2026-10-30 13:30:04.582286808 +0000 UTC m=+2646.165716495"
Oct 30 13:30:04.782434 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:04.780113    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"]
Oct 30 13:30:04.818260 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:30:04.818227    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400e5f9c_8609_4854_af56_2f31915c6f13.slice/crio-7b452fe774340dd7fafbf3c5ba4d81e1b066f07c1493ae535f2accb511e8767b WatchSource:0}: Error finding container 7b452fe774340dd7fafbf3c5ba4d81e1b066f07c1493ae535f2accb511e8767b: Status 404 returned error can't find the container with id 7b452fe774340dd7fafbf3c5ba4d81e1b066f07c1493ae535f2accb511e8767b
Oct 30 13:30:05.331902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.331871    5893 generic.go:356] "Generic (PLEG): container finished" podID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2" containerID="33f43ea2de0a20cec0498b91cc19017b9ccf8554243f97bfd4e98170189f2c71" exitCode=60
Oct 30 13:30:05.332400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.331917    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2" event={"ID":"871e8c5b-99ee-48f1-8a0a-ae1846253ec2","Type":"ContainerDied","Data":"33f43ea2de0a20cec0498b91cc19017b9ccf8554243f97bfd4e98170189f2c71"}
Oct 30 13:30:05.332400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.331977    5893 scope.go:122] "RemoveContainer" containerID="e9d4440c837161a38fe4a650b226bbdd30e32765458c401dbd382a841b694979"
Oct 30 13:30:05.332400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.332323    5893 scope.go:122] "RemoveContainer" containerID="33f43ea2de0a20cec0498b91cc19017b9ccf8554243f97bfd4e98170189f2c71"
Oct 30 13:30:05.332676 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:05.332654    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2_odh-ai-gateway-infra(871e8c5b-99ee-48f1-8a0a-ae1846253ec2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2" podUID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2"
Oct 30 13:30:05.339360 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.338900    5893 generic.go:356] "Generic (PLEG): container finished" podID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a" containerID="9cf9e815ee1a749adf7779b2eac7ff002ff0905088cb55e5f86cef6154f0f9e8" exitCode=60
Oct 30 13:30:05.339491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.339375    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd" event={"ID":"78c9276f-5e11-4866-8a30-1bb4c38a5e8a","Type":"ContainerDied","Data":"9cf9e815ee1a749adf7779b2eac7ff002ff0905088cb55e5f86cef6154f0f9e8"}
Oct 30 13:30:05.339491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.339413    5893 scope.go:122] "RemoveContainer" containerID="9cf9e815ee1a749adf7779b2eac7ff002ff0905088cb55e5f86cef6154f0f9e8"
Oct 30 13:30:05.339809 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:05.339766    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd_odh-ai-gateway-infra(78c9276f-5e11-4866-8a30-1bb4c38a5e8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd" podUID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a"
Oct 30 13:30:05.348428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.345503    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc" event={"ID":"400e5f9c-8609-4854-af56-2f31915c6f13","Type":"ContainerStarted","Data":"7b452fe774340dd7fafbf3c5ba4d81e1b066f07c1493ae535f2accb511e8767b"}
Oct 30 13:30:05.366782 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.359459    5893 generic.go:356] "Generic (PLEG): container finished" podID="c9484142-6e89-4dc7-89ff-0fcd438cfedb" containerID="327d5a027a9a190aea0c41daf46d9bee83ee4650eb7fefe1eb5f1555adf6c47d" exitCode=60
Oct 30 13:30:05.366782 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.359612    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk" event={"ID":"c9484142-6e89-4dc7-89ff-0fcd438cfedb","Type":"ContainerDied","Data":"327d5a027a9a190aea0c41daf46d9bee83ee4650eb7fefe1eb5f1555adf6c47d"}
Oct 30 13:30:05.366782 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.360207    5893 scope.go:122] "RemoveContainer" containerID="327d5a027a9a190aea0c41daf46d9bee83ee4650eb7fefe1eb5f1555adf6c47d"
Oct 30 13:30:05.380409 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.376579    5893 generic.go:356] "Generic (PLEG): container finished" podID="8d60b5df-0998-4618-b201-4793314d793d" containerID="3c9adad7e5ad0f04471a62f656fa0e10823799b4c7578367fa7011ef2de76feb" exitCode=60
Oct 30 13:30:05.380409 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.376645    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r" event={"ID":"8d60b5df-0998-4618-b201-4793314d793d","Type":"ContainerDied","Data":"3c9adad7e5ad0f04471a62f656fa0e10823799b4c7578367fa7011ef2de76feb"}
Oct 30 13:30:05.380409 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.377164    5893 scope.go:122] "RemoveContainer" containerID="3c9adad7e5ad0f04471a62f656fa0e10823799b4c7578367fa7011ef2de76feb"
Oct 30 13:30:05.380409 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:05.377633    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r_odh-ai-gateway-infra(8d60b5df-0998-4618-b201-4793314d793d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r" podUID="8d60b5df-0998-4618-b201-4793314d793d"
Oct 30 13:30:05.380409 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:05.378797    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk_odh-ai-gateway-infra(c9484142-6e89-4dc7-89ff-0fcd438cfedb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk" podUID="c9484142-6e89-4dc7-89ff-0fcd438cfedb"
Oct 30 13:30:05.396587 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.396555    5893 scope.go:122] "RemoveContainer" containerID="cc526fdb708f8ab719010b7d57245074e8d548855942c2c4b69f4c6d47ba1918"
Oct 30 13:30:05.421687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.421660    5893 scope.go:122] "RemoveContainer" containerID="df1ee42222cf0c1e12af316aee8c8b1d9277420eca3ea3e35329af7842708a65"
Oct 30 13:30:05.481469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:05.481444    5893 scope.go:122] "RemoveContainer" containerID="93052a7e508eadd5c2332c8b56759ccc1c56b2919b0080de9b57c6902f8e9055"
Oct 30 13:30:06.126077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.126038    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"]
Oct 30 13:30:06.133324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.133302    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.187347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.180699    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-03b5b3-openshift-default-dockercfg-dp4wq\""
Oct 30 13:30:06.202632 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.202598    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"]
Oct 30 13:30:06.322293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.322227    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqq6v\" (UniqueName: \"kubernetes.io/projected/84c68bf3-ee6a-498f-aeb1-0318f788488d-kube-api-access-dqq6v\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.327473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.327440    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.348472 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.348431    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.349114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.349090    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c68bf3-ee6a-498f-aeb1-0318f788488d-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.349365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.349346    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.349514 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.349499    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.349690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.349669    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.349877 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.349863    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.350056 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.350036    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.431358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.430878    5893 scope.go:122] "RemoveContainer" containerID="33f43ea2de0a20cec0498b91cc19017b9ccf8554243f97bfd4e98170189f2c71"
Oct 30 13:30:06.431553 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:06.431354    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2_odh-ai-gateway-infra(871e8c5b-99ee-48f1-8a0a-ae1846253ec2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2" podUID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2"
Oct 30 13:30:06.435295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.434722    5893 scope.go:122] "RemoveContainer" containerID="9cf9e815ee1a749adf7779b2eac7ff002ff0905088cb55e5f86cef6154f0f9e8"
Oct 30 13:30:06.435295 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:06.435130    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd_odh-ai-gateway-infra(78c9276f-5e11-4866-8a30-1bb4c38a5e8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd" podUID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a"
Oct 30 13:30:06.442297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.437569    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc" event={"ID":"400e5f9c-8609-4854-af56-2f31915c6f13","Type":"ContainerStarted","Data":"339df0ccd3cfd2845eac65a833d72b05ee0a10727d2f49afdd636a453bd31e96"}
Oct 30 13:30:06.442297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.438387    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:06.442297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.441155    5893 scope.go:122] "RemoveContainer" containerID="327d5a027a9a190aea0c41daf46d9bee83ee4650eb7fefe1eb5f1555adf6c47d"
Oct 30 13:30:06.442297 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:06.441514    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk_odh-ai-gateway-infra(c9484142-6e89-4dc7-89ff-0fcd438cfedb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk" podUID="c9484142-6e89-4dc7-89ff-0fcd438cfedb"
Oct 30 13:30:06.456324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.443982    5893 scope.go:122] "RemoveContainer" containerID="3c9adad7e5ad0f04471a62f656fa0e10823799b4c7578367fa7011ef2de76feb"
Oct 30 13:30:06.456324 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:06.444314    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r_odh-ai-gateway-infra(8d60b5df-0998-4618-b201-4793314d793d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r" podUID="8d60b5df-0998-4618-b201-4793314d793d"
Oct 30 13:30:06.456324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.454063    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.456324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.454113    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.456324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.454146    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.456324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.454242    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.456324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.454325    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.456324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.454421    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqq6v\" (UniqueName: \"kubernetes.io/projected/84c68bf3-ee6a-498f-aeb1-0318f788488d-kube-api-access-dqq6v\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.456324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.454537    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.456324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.454583    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.456324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.454719    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c68bf3-ee6a-498f-aeb1-0318f788488d-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.457051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.457036    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.460291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.457696    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.460291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.457914    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.460291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.460073    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.460291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.460259    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c68bf3-ee6a-498f-aeb1-0318f788488d-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.485300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.479137    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.494339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.492687    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.496697 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.496666    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqq6v\" (UniqueName: \"kubernetes.io/projected/84c68bf3-ee6a-498f-aeb1-0318f788488d-kube-api-access-dqq6v\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.498053 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.497642    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:06.539518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.539472    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc" podStartSLOduration=3.034645007 podStartE2EDuration="3.539457884s" totalImagesPullingTime="504.812877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:06.537902142 +0000 UTC m=+2648.121331835" watchObservedRunningTime="2026-10-30 13:30:06.539457884 +0000 UTC m=+2648.122887576"
Oct 30 13:30:06.804503 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:06.794584    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:07.190245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:07.190175    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"]
Oct 30 13:30:07.190436 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:30:07.190336    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84c68bf3_ee6a_498f_aeb1_0318f788488d.slice/crio-70be24072e078e07c60de020e8404b9962e2c67953fdf6b82bc9f4ff62404fe1 WatchSource:0}: Error finding container 70be24072e078e07c60de020e8404b9962e2c67953fdf6b82bc9f4ff62404fe1: Status 404 returned error can't find the container with id 70be24072e078e07c60de020e8404b9962e2c67953fdf6b82bc9f4ff62404fe1
Oct 30 13:30:07.193761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:07.193729    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:07.193897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:07.193793    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:07.193897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:07.193827    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:07.483851 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:07.483114    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" event={"ID":"84c68bf3-ee6a-498f-aeb1-0318f788488d","Type":"ContainerStarted","Data":"c26c2dff88c0c911ef788fb28fb3c7c1be58d6589d6f694aa062aeaabbc6ddab"}
Oct 30 13:30:07.483851 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:07.483167    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" event={"ID":"84c68bf3-ee6a-498f-aeb1-0318f788488d","Type":"ContainerStarted","Data":"70be24072e078e07c60de020e8404b9962e2c67953fdf6b82bc9f4ff62404fe1"}
Oct 30 13:30:07.537644 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:07.537596    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" podStartSLOduration=1.537581629 podStartE2EDuration="1.537581629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:07.534226124 +0000 UTC m=+2649.117655817" watchObservedRunningTime="2026-10-30 13:30:07.537581629 +0000 UTC m=+2649.121011387"
Oct 30 13:30:07.827420 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:07.821377    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:08.807801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:08.807766    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:09.504436 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:09.504408    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:09.505703 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:09.505681    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:30:11.952443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:11.952410    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"]
Oct 30 13:30:11.974675 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:11.974491    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"
Oct 30 13:30:11.981295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:11.981249    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"]
Oct 30 13:30:12.002180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.002091    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-03b5b3-dockercfg-7hwl2\""
Oct 30 13:30:12.002180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.002152    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-03b5b3\""
Oct 30 13:30:12.051296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.041494    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"]
Oct 30 13:30:12.161868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.120952    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"
Oct 30 13:30:12.161868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.135376    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x7lr\" (UniqueName: \"kubernetes.io/projected/2b81d985-3126-4e47-b9f9-6e9665d7dff0-kube-api-access-8x7lr\") pod \"payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p\" (UID: \"2b81d985-3126-4e47-b9f9-6e9665d7dff0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"
Oct 30 13:30:12.161868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.135458    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znj2t\" (UniqueName: \"kubernetes.io/projected/d445892f-5fd7-42a2-ae10-f779e2e37dfa-kube-api-access-znj2t\") pod \"payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn\" (UID: \"d445892f-5fd7-42a2-ae10-f779e2e37dfa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"
Oct 30 13:30:12.161868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.135533    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d445892f-5fd7-42a2-ae10-f779e2e37dfa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn\" (UID: \"d445892f-5fd7-42a2-ae10-f779e2e37dfa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"
Oct 30 13:30:12.161868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.135585    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b81d985-3126-4e47-b9f9-6e9665d7dff0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p\" (UID: \"2b81d985-3126-4e47-b9f9-6e9665d7dff0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"
Oct 30 13:30:12.169300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.164116    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"]
Oct 30 13:30:12.253460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.253355    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x7lr\" (UniqueName: \"kubernetes.io/projected/2b81d985-3126-4e47-b9f9-6e9665d7dff0-kube-api-access-8x7lr\") pod \"payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p\" (UID: \"2b81d985-3126-4e47-b9f9-6e9665d7dff0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"
Oct 30 13:30:12.253643 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.253570    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znj2t\" (UniqueName: \"kubernetes.io/projected/d445892f-5fd7-42a2-ae10-f779e2e37dfa-kube-api-access-znj2t\") pod \"payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn\" (UID: \"d445892f-5fd7-42a2-ae10-f779e2e37dfa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"
Oct 30 13:30:12.253721 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.253684    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d445892f-5fd7-42a2-ae10-f779e2e37dfa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn\" (UID: \"d445892f-5fd7-42a2-ae10-f779e2e37dfa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"
Oct 30 13:30:12.253780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.253761    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b81d985-3126-4e47-b9f9-6e9665d7dff0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p\" (UID: \"2b81d985-3126-4e47-b9f9-6e9665d7dff0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"
Oct 30 13:30:12.254995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.254961    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b81d985-3126-4e47-b9f9-6e9665d7dff0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p\" (UID: \"2b81d985-3126-4e47-b9f9-6e9665d7dff0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"
Oct 30 13:30:12.256502 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.256472    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d445892f-5fd7-42a2-ae10-f779e2e37dfa-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn\" (UID: \"d445892f-5fd7-42a2-ae10-f779e2e37dfa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"
Oct 30 13:30:12.285121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.284532    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znj2t\" (UniqueName: \"kubernetes.io/projected/d445892f-5fd7-42a2-ae10-f779e2e37dfa-kube-api-access-znj2t\") pod \"payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn\" (UID: \"d445892f-5fd7-42a2-ae10-f779e2e37dfa\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"
Oct 30 13:30:12.289793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.289759    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x7lr\" (UniqueName: \"kubernetes.io/projected/2b81d985-3126-4e47-b9f9-6e9665d7dff0-kube-api-access-8x7lr\") pod \"payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p\" (UID: \"2b81d985-3126-4e47-b9f9-6e9665d7dff0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"
Oct 30 13:30:12.297873 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.297651    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"
Oct 30 13:30:12.321512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.321486    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"
Oct 30 13:30:12.786515 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.678852    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"]
Oct 30 13:30:12.786515 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:30:12.683790    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd445892f_5fd7_42a2_ae10_f779e2e37dfa.slice/crio-2d9a77b67cec72bd93422f84237c1de09f4018716ee77ceab0746cdd2296728d WatchSource:0}: Error finding container 2d9a77b67cec72bd93422f84237c1de09f4018716ee77ceab0746cdd2296728d: Status 404 returned error can't find the container with id 2d9a77b67cec72bd93422f84237c1de09f4018716ee77ceab0746cdd2296728d
Oct 30 13:30:12.786515 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:30:12.692546    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b81d985_3126_4e47_b9f9_6e9665d7dff0.slice/crio-50f498ad5a87e6449471e00715acba683993d20d9f2e9d906240d34eeb169fdf WatchSource:0}: Error finding container 50f498ad5a87e6449471e00715acba683993d20d9f2e9d906240d34eeb169fdf: Status 404 returned error can't find the container with id 50f498ad5a87e6449471e00715acba683993d20d9f2e9d906240d34eeb169fdf
Oct 30 13:30:12.786515 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.692822    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"]
Oct 30 13:30:12.894637 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.894600    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:30:12.894813 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:12.894653    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:30:13.504884 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.504840    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:30:13.544459 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.544411    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p" event={"ID":"2b81d985-3126-4e47-b9f9-6e9665d7dff0","Type":"ContainerStarted","Data":"d1d68779b685da760413e427b6aeb775889d18c1b94b01a8d67180f074617228"}
Oct 30 13:30:13.544655 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.544478    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p" event={"ID":"2b81d985-3126-4e47-b9f9-6e9665d7dff0","Type":"ContainerStarted","Data":"50f498ad5a87e6449471e00715acba683993d20d9f2e9d906240d34eeb169fdf"}
Oct 30 13:30:13.544655 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.544619    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"
Oct 30 13:30:13.546709 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.546680    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn" event={"ID":"d445892f-5fd7-42a2-ae10-f779e2e37dfa","Type":"ContainerStarted","Data":"08f81ac6824bc23ab0284348259605095988dc77806772bb3d3a99ea6699b385"}
Oct 30 13:30:13.546874 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.546734    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn" event={"ID":"d445892f-5fd7-42a2-ae10-f779e2e37dfa","Type":"ContainerStarted","Data":"2d9a77b67cec72bd93422f84237c1de09f4018716ee77ceab0746cdd2296728d"}
Oct 30 13:30:13.546874 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.546766    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"
Oct 30 13:30:13.549411 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.549384    5893 generic.go:356] "Generic (PLEG): container finished" podID="64c1d503-512b-4521-af3f-668e94ed5b99" containerID="214caddcae79781a68b6918a19ba6a299bf15c42a3f346f857ba5868515234e5" exitCode=0
Oct 30 13:30:13.549588 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.549451    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" event={"ID":"64c1d503-512b-4521-af3f-668e94ed5b99","Type":"ContainerDied","Data":"214caddcae79781a68b6918a19ba6a299bf15c42a3f346f857ba5868515234e5"}
Oct 30 13:30:13.550056 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.549837    5893 scope.go:122] "RemoveContainer" containerID="214caddcae79781a68b6918a19ba6a299bf15c42a3f346f857ba5868515234e5"
Oct 30 13:30:13.552902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.552875    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:13.553029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.553008    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:13.553131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.553064    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:13.653947 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.653877    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn" podStartSLOduration=1.653856267 podStartE2EDuration="1.653856267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:13.651761139 +0000 UTC m=+2655.235190833" watchObservedRunningTime="2026-10-30 13:30:13.653856267 +0000 UTC m=+2655.237285961"
Oct 30 13:30:13.694329 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:13.694255    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p" podStartSLOduration=2.694242745 podStartE2EDuration="2.694242745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:13.693610869 +0000 UTC m=+2655.277040562" watchObservedRunningTime="2026-10-30 13:30:13.694242745 +0000 UTC m=+2655.277672438"
Oct 30 13:30:14.580454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:14.580417    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" event={"ID":"64c1d503-512b-4521-af3f-668e94ed5b99","Type":"ContainerStarted","Data":"47ebcee4149f5122d1f56ec827bf0551e474652c23fa8a02a5d075f0a55c8dcd"}
Oct 30 13:30:15.135167 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:15.135135    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:30:15.135360 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:15.135314    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:30:15.135414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:15.135363    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 13:30:15.524219 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:15.524184    5893 scope.go:122] "RemoveContainer" containerID="9cf9e815ee1a749adf7779b2eac7ff002ff0905088cb55e5f86cef6154f0f9e8"
Oct 30 13:30:16.135073 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:16.134996    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:30:16.135073 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:16.135049    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 13:30:16.524598 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:16.524566    5893 scope.go:122] "RemoveContainer" containerID="3c9adad7e5ad0f04471a62f656fa0e10823799b4c7578367fa7011ef2de76feb"
Oct 30 13:30:16.524794 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:16.524756    5893 scope.go:122] "RemoveContainer" containerID="33f43ea2de0a20cec0498b91cc19017b9ccf8554243f97bfd4e98170189f2c71"
Oct 30 13:30:16.525069 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:16.525048    5893 scope.go:122] "RemoveContainer" containerID="327d5a027a9a190aea0c41daf46d9bee83ee4650eb7fefe1eb5f1555adf6c47d"
Oct 30 13:30:16.607390 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:16.607332    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd" event={"ID":"78c9276f-5e11-4866-8a30-1bb4c38a5e8a","Type":"ContainerStarted","Data":"e6a7ee7849355bd67bf09cbb7927816721b959f69ea66ec57ebecbdb3a782a36"}
Oct 30 13:30:17.137289 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:17.136045    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:30:17.137289 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:17.136105    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 13:30:17.619801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:17.619753    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk" event={"ID":"c9484142-6e89-4dc7-89ff-0fcd438cfedb","Type":"ContainerStarted","Data":"22a68d4b91436a635c6e9f4c62d17c9bcee587c3e6a41c60d6cb1f52aeaed122"}
Oct 30 13:30:17.622439 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:17.622413    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r" event={"ID":"8d60b5df-0998-4618-b201-4793314d793d","Type":"ContainerStarted","Data":"cb6ad306956961cd01fe95a9c0cc7975f26057a4e1ff0ec1ee95b78ececb1b45"}
Oct 30 13:30:17.624620 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:17.624601    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2" event={"ID":"871e8c5b-99ee-48f1-8a0a-ae1846253ec2","Type":"ContainerStarted","Data":"889b46f96df55938d32ca9271320feb76933dc2cbb6b3ab1b88cb6bb4444542f"}
Oct 30 13:30:17.627234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:17.627209    5893 generic.go:356] "Generic (PLEG): container finished" podID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a" containerID="e6a7ee7849355bd67bf09cbb7927816721b959f69ea66ec57ebecbdb3a782a36" exitCode=60
Oct 30 13:30:17.627382 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:17.627245    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd" event={"ID":"78c9276f-5e11-4866-8a30-1bb4c38a5e8a","Type":"ContainerDied","Data":"e6a7ee7849355bd67bf09cbb7927816721b959f69ea66ec57ebecbdb3a782a36"}
Oct 30 13:30:17.627382 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:17.627301    5893 scope.go:122] "RemoveContainer" containerID="9cf9e815ee1a749adf7779b2eac7ff002ff0905088cb55e5f86cef6154f0f9e8"
Oct 30 13:30:17.627695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:17.627677    5893 scope.go:122] "RemoveContainer" containerID="e6a7ee7849355bd67bf09cbb7927816721b959f69ea66ec57ebecbdb3a782a36"
Oct 30 13:30:17.627951 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:17.627930    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd_odh-ai-gateway-infra(78c9276f-5e11-4866-8a30-1bb4c38a5e8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd" podUID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a"
Oct 30 13:30:17.688958 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:17.688924    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"]
Oct 30 13:30:18.135032 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.134996    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:30:18.135218 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.135062    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 13:30:18.262219 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.262153    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"]
Oct 30 13:30:18.278391 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.275025    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.286210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.285613    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"]
Oct 30 13:30:18.297805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.289588    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-96ab73-openshift-default-dockercfg-nshzl\""
Oct 30 13:30:18.397032 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.396995    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-workload-certs\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.397402 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.397327    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-istio-envoy\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.397623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.397605    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c3dae9-4165-4660-bff5-f728025903cd-istio-podinfo\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.397813 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.397797    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c3dae9-4165-4660-bff5-f728025903cd-istio-token\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.398054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.398039    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-workload-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.398239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.398218    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c3dae9-4165-4660-bff5-f728025903cd-istiod-ca-cert\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.398416 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.398287    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgttl\" (UniqueName: \"kubernetes.io/projected/91c3dae9-4165-4660-bff5-f728025903cd-kube-api-access-qgttl\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.398416 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.398363    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-credential-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.398416 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.398408    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-istio-data\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.501795 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.501753    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-workload-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.501990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.501874    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c3dae9-4165-4660-bff5-f728025903cd-istiod-ca-cert\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.501990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.501922    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgttl\" (UniqueName: \"kubernetes.io/projected/91c3dae9-4165-4660-bff5-f728025903cd-kube-api-access-qgttl\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.501990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.501951    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-credential-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.501990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.501986    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-istio-data\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.502252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.502130    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-workload-certs\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.502252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.502173    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-istio-envoy\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.502252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.502248    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c3dae9-4165-4660-bff5-f728025903cd-istio-podinfo\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.502536 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.502290    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c3dae9-4165-4660-bff5-f728025903cd-istio-token\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.503141 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.503115    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-workload-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.503957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.503933    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c3dae9-4165-4660-bff5-f728025903cd-istiod-ca-cert\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.504513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.504488    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-credential-socket\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.504761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.504742    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-istio-data\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.506862 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.506830    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-workload-certs\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.507836 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.507781    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-istio-envoy\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.509501 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.509475    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c3dae9-4165-4660-bff5-f728025903cd-istio-podinfo\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.573986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.573915    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c3dae9-4165-4660-bff5-f728025903cd-istio-token\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.574670 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.574643    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgttl\" (UniqueName: \"kubernetes.io/projected/91c3dae9-4165-4660-bff5-f728025903cd-kube-api-access-qgttl\") pod \"e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") " pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.600791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.600765    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:18.663402 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.663361    5893 generic.go:356] "Generic (PLEG): container finished" podID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2" containerID="889b46f96df55938d32ca9271320feb76933dc2cbb6b3ab1b88cb6bb4444542f" exitCode=60
Oct 30 13:30:18.663803 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.663441    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2" event={"ID":"871e8c5b-99ee-48f1-8a0a-ae1846253ec2","Type":"ContainerDied","Data":"889b46f96df55938d32ca9271320feb76933dc2cbb6b3ab1b88cb6bb4444542f"}
Oct 30 13:30:18.664610 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.664579    5893 scope.go:122] "RemoveContainer" containerID="33f43ea2de0a20cec0498b91cc19017b9ccf8554243f97bfd4e98170189f2c71"
Oct 30 13:30:18.665629 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.665613    5893 scope.go:122] "RemoveContainer" containerID="889b46f96df55938d32ca9271320feb76933dc2cbb6b3ab1b88cb6bb4444542f"
Oct 30 13:30:18.666873 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:18.666534    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2_odh-ai-gateway-infra(871e8c5b-99ee-48f1-8a0a-ae1846253ec2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2" podUID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2"
Oct 30 13:30:18.681770 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.680386    5893 generic.go:356] "Generic (PLEG): container finished" podID="c9484142-6e89-4dc7-89ff-0fcd438cfedb" containerID="22a68d4b91436a635c6e9f4c62d17c9bcee587c3e6a41c60d6cb1f52aeaed122" exitCode=60
Oct 30 13:30:18.681770 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.680537    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk" event={"ID":"c9484142-6e89-4dc7-89ff-0fcd438cfedb","Type":"ContainerDied","Data":"22a68d4b91436a635c6e9f4c62d17c9bcee587c3e6a41c60d6cb1f52aeaed122"}
Oct 30 13:30:18.681770 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.681013    5893 scope.go:122] "RemoveContainer" containerID="22a68d4b91436a635c6e9f4c62d17c9bcee587c3e6a41c60d6cb1f52aeaed122"
Oct 30 13:30:18.681770 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:18.681483    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk_odh-ai-gateway-infra(c9484142-6e89-4dc7-89ff-0fcd438cfedb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk" podUID="c9484142-6e89-4dc7-89ff-0fcd438cfedb"
Oct 30 13:30:18.684582 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.684554    5893 generic.go:356] "Generic (PLEG): container finished" podID="8d60b5df-0998-4618-b201-4793314d793d" containerID="cb6ad306956961cd01fe95a9c0cc7975f26057a4e1ff0ec1ee95b78ececb1b45" exitCode=60
Oct 30 13:30:18.684691 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.684627    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r" event={"ID":"8d60b5df-0998-4618-b201-4793314d793d","Type":"ContainerDied","Data":"cb6ad306956961cd01fe95a9c0cc7975f26057a4e1ff0ec1ee95b78ececb1b45"}
Oct 30 13:30:18.685210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.685170    5893 scope.go:122] "RemoveContainer" containerID="cb6ad306956961cd01fe95a9c0cc7975f26057a4e1ff0ec1ee95b78ececb1b45"
Oct 30 13:30:18.685712 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:18.685684    5893 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r_odh-ai-gateway-infra(8d60b5df-0998-4618-b201-4793314d793d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r" podUID="8d60b5df-0998-4618-b201-4793314d793d"
Oct 30 13:30:18.697509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.696829    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"]
Oct 30 13:30:18.712180 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:18.712148    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:30:18.729747 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.729705    5893 scope.go:122] "RemoveContainer" containerID="327d5a027a9a190aea0c41daf46d9bee83ee4650eb7fefe1eb5f1555adf6c47d"
Oct 30 13:30:18.807151 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.807033    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"]
Oct 30 13:30:18.893424 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.893403    5893 scope.go:122] "RemoveContainer" containerID="3c9adad7e5ad0f04471a62f656fa0e10823799b4c7578367fa7011ef2de76feb"
Oct 30 13:30:18.895741 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:18.895666    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"]
Oct 30 13:30:19.063921 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.063863    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"]
Oct 30 13:30:19.119141 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.119096    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:19.119293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.119192    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:19.119293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.119240    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:19.134536 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.134505    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Oct 30 13:30:19.134745 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.134718    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Oct 30 13:30:19.297573 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.297484    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"
Oct 30 13:30:19.396342 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.394134    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78c9276f-5e11-4866-8a30-1bb4c38a5e8a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9276f-5e11-4866-8a30-1bb4c38a5e8a-maas-api-service-ca\") pod \"78c9276f-5e11-4866-8a30-1bb4c38a5e8a\" (UID: \"78c9276f-5e11-4866-8a30-1bb4c38a5e8a\") "
Oct 30 13:30:19.396342 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.394192    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78c9276f-5e11-4866-8a30-1bb4c38a5e8a-kube-api-access-5bmps\" (UniqueName: \"kubernetes.io/projected/78c9276f-5e11-4866-8a30-1bb4c38a5e8a-kube-api-access-5bmps\") pod \"78c9276f-5e11-4866-8a30-1bb4c38a5e8a\" (UID: \"78c9276f-5e11-4866-8a30-1bb4c38a5e8a\") "
Oct 30 13:30:19.400125 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.397647    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c9276f-5e11-4866-8a30-1bb4c38a5e8a-maas-api-service-ca" pod "78c9276f-5e11-4866-8a30-1bb4c38a5e8a" (UID: "78c9276f-5e11-4866-8a30-1bb4c38a5e8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:19.411639 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.411603    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c9276f-5e11-4866-8a30-1bb4c38a5e8a-kube-api-access-5bmps" pod "78c9276f-5e11-4866-8a30-1bb4c38a5e8a" (UID: "78c9276f-5e11-4866-8a30-1bb4c38a5e8a"). InnerVolumeSpecName "kube-api-access-5bmps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:19.500318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.497066    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78c9276f-5e11-4866-8a30-1bb4c38a5e8a-maas-api-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:19.500318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.497101    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bmps\" (UniqueName: \"kubernetes.io/projected/78c9276f-5e11-4866-8a30-1bb4c38a5e8a-kube-api-access-5bmps\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:19.698533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.698326    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd" event={"ID":"78c9276f-5e11-4866-8a30-1bb4c38a5e8a","Type":"ContainerDied","Data":"77623209bb763b84ddd27e45320e97ce3d5090833d36b95d5fbd9d9dd46fbe03"}
Oct 30 13:30:19.698533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.698352    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"
Oct 30 13:30:19.698533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.698401    5893 scope.go:122] "RemoveContainer" containerID="e6a7ee7849355bd67bf09cbb7927816721b959f69ea66ec57ebecbdb3a782a36"
Oct 30 13:30:19.717419 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.717395    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn" event={"ID":"91c3dae9-4165-4660-bff5-f728025903cd","Type":"ContainerStarted","Data":"c1a79b3af28d0b470e7871b0474881f3c3512f5d3adba73c351966fd473f7504"}
Oct 30 13:30:19.718084 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.717636    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn" event={"ID":"91c3dae9-4165-4660-bff5-f728025903cd","Type":"ContainerStarted","Data":"13fdfc9cda652e0c98d87c741d10dd1dc192ca4a5e0a9597a718b7be7cac2023"}
Oct 30 13:30:19.943395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.943359    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"]
Oct 30 13:30:19.964120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:19.964086    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-dd90cb-29889450-gmkkd"]
Oct 30 13:30:20.166869 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.166840    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"
Oct 30 13:30:20.216162 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.216079    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"
Oct 30 13:30:20.219461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.219427    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/871e8c5b-99ee-48f1-8a0a-ae1846253ec2-kube-api-access-ppdmd\" (UniqueName: \"kubernetes.io/projected/871e8c5b-99ee-48f1-8a0a-ae1846253ec2-kube-api-access-ppdmd\") pod \"871e8c5b-99ee-48f1-8a0a-ae1846253ec2\" (UID: \"871e8c5b-99ee-48f1-8a0a-ae1846253ec2\") "
Oct 30 13:30:20.219625 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.219480    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d60b5df-0998-4618-b201-4793314d793d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d60b5df-0998-4618-b201-4793314d793d-maas-api-service-ca\") pod \"8d60b5df-0998-4618-b201-4793314d793d\" (UID: \"8d60b5df-0998-4618-b201-4793314d793d\") "
Oct 30 13:30:20.219711 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.219645    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d60b5df-0998-4618-b201-4793314d793d-kube-api-access-jnw8m\" (UniqueName: \"kubernetes.io/projected/8d60b5df-0998-4618-b201-4793314d793d-kube-api-access-jnw8m\") pod \"8d60b5df-0998-4618-b201-4793314d793d\" (UID: \"8d60b5df-0998-4618-b201-4793314d793d\") "
Oct 30 13:30:20.219711 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.219688    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/871e8c5b-99ee-48f1-8a0a-ae1846253ec2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871e8c5b-99ee-48f1-8a0a-ae1846253ec2-maas-api-service-ca\") pod \"871e8c5b-99ee-48f1-8a0a-ae1846253ec2\" (UID: \"871e8c5b-99ee-48f1-8a0a-ae1846253ec2\") "
Oct 30 13:30:20.220012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.219981    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"
Oct 30 13:30:20.221823 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.221797    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d60b5df-0998-4618-b201-4793314d793d-maas-api-service-ca" pod "8d60b5df-0998-4618-b201-4793314d793d" (UID: "8d60b5df-0998-4618-b201-4793314d793d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:20.222735 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.222710    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871e8c5b-99ee-48f1-8a0a-ae1846253ec2-maas-api-service-ca" pod "871e8c5b-99ee-48f1-8a0a-ae1846253ec2" (UID: "871e8c5b-99ee-48f1-8a0a-ae1846253ec2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:20.224791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.224766    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871e8c5b-99ee-48f1-8a0a-ae1846253ec2-kube-api-access-ppdmd" pod "871e8c5b-99ee-48f1-8a0a-ae1846253ec2" (UID: "871e8c5b-99ee-48f1-8a0a-ae1846253ec2"). InnerVolumeSpecName "kube-api-access-ppdmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:20.225473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.225448    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d60b5df-0998-4618-b201-4793314d793d-kube-api-access-jnw8m" pod "8d60b5df-0998-4618-b201-4793314d793d" (UID: "8d60b5df-0998-4618-b201-4793314d793d"). InnerVolumeSpecName "kube-api-access-jnw8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:20.257344 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.255719    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:30:20.257590 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.257571    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:30:20.298778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.293112    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn" podStartSLOduration=2.293088347 podStartE2EDuration="2.293088347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:20.02473274 +0000 UTC m=+2661.608162434" watchObservedRunningTime="2026-10-30 13:30:20.293088347 +0000 UTC m=+2661.876518041"
Oct 30 13:30:20.299590 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.299565    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:30:20.321724 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.321686    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9484142-6e89-4dc7-89ff-0fcd438cfedb-kube-api-access-8swbp\" (UniqueName: \"kubernetes.io/projected/c9484142-6e89-4dc7-89ff-0fcd438cfedb-kube-api-access-8swbp\") pod \"c9484142-6e89-4dc7-89ff-0fcd438cfedb\" (UID: \"c9484142-6e89-4dc7-89ff-0fcd438cfedb\") "
Oct 30 13:30:20.322093 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.322075    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9484142-6e89-4dc7-89ff-0fcd438cfedb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9484142-6e89-4dc7-89ff-0fcd438cfedb-maas-api-service-ca\") pod \"c9484142-6e89-4dc7-89ff-0fcd438cfedb\" (UID: \"c9484142-6e89-4dc7-89ff-0fcd438cfedb\") "
Oct 30 13:30:20.322931 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.322907    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppdmd\" (UniqueName: \"kubernetes.io/projected/871e8c5b-99ee-48f1-8a0a-ae1846253ec2-kube-api-access-ppdmd\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:20.323071 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.323057    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d60b5df-0998-4618-b201-4793314d793d-maas-api-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:20.323242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.323230    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnw8m\" (UniqueName: \"kubernetes.io/projected/8d60b5df-0998-4618-b201-4793314d793d-kube-api-access-jnw8m\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:20.323361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.323349    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/871e8c5b-99ee-48f1-8a0a-ae1846253ec2-maas-api-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:20.323783 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.323762    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9484142-6e89-4dc7-89ff-0fcd438cfedb-maas-api-service-ca" pod "c9484142-6e89-4dc7-89ff-0fcd438cfedb" (UID: "c9484142-6e89-4dc7-89ff-0fcd438cfedb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:20.331870 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.331835    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9484142-6e89-4dc7-89ff-0fcd438cfedb-kube-api-access-8swbp" pod "c9484142-6e89-4dc7-89ff-0fcd438cfedb" (UID: "c9484142-6e89-4dc7-89ff-0fcd438cfedb"). InnerVolumeSpecName "kube-api-access-8swbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:20.425005 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.424971    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9484142-6e89-4dc7-89ff-0fcd438cfedb-maas-api-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:20.425244 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.425225    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8swbp\" (UniqueName: \"kubernetes.io/projected/c9484142-6e89-4dc7-89ff-0fcd438cfedb-kube-api-access-8swbp\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:20.581203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.581110    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a" path="/var/lib/kubelet/pods/78c9276f-5e11-4866-8a30-1bb4c38a5e8a/volumes"
Oct 30 13:30:20.601614 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.601569    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:20.603124 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.603094    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:30:20.603306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.603139    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn" podUID="91c3dae9-4165-4660-bff5-f728025903cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 13:30:20.734911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.734870    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk" event={"ID":"c9484142-6e89-4dc7-89ff-0fcd438cfedb","Type":"ContainerDied","Data":"ec2a77c13a44f83561ca4e5c82ce101b5ec442e9e429fb1c16302ee8ee23032b"}
Oct 30 13:30:20.735123 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.734988    5893 scope.go:122] "RemoveContainer" containerID="22a68d4b91436a635c6e9f4c62d17c9bcee587c3e6a41c60d6cb1f52aeaed122"
Oct 30 13:30:20.735197 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.735169    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"
Oct 30 13:30:20.768722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.768678    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r" event={"ID":"8d60b5df-0998-4618-b201-4793314d793d","Type":"ContainerDied","Data":"3305f6a54cd9af2215a8adab575b76a5d76a845b41a609dcc272f973370628c3"}
Oct 30 13:30:20.768942 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.768866    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"
Oct 30 13:30:20.772182 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.772149    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2" event={"ID":"871e8c5b-99ee-48f1-8a0a-ae1846253ec2","Type":"ContainerDied","Data":"eea49000e00fda0fa5f91df55e72ccb68fd441b43f7e63b4cc10fedac5c843b6"}
Oct 30 13:30:20.772384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.772329    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"
Oct 30 13:30:20.797427 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.797407    5893 scope.go:122] "RemoveContainer" containerID="cb6ad306956961cd01fe95a9c0cc7975f26057a4e1ff0ec1ee95b78ececb1b45"
Oct 30 13:30:20.801731 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.801702    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"]
Oct 30 13:30:20.808561 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.808544    5893 scope.go:122] "RemoveContainer" containerID="889b46f96df55938d32ca9271320feb76933dc2cbb6b3ab1b88cb6bb4444542f"
Oct 30 13:30:20.812737 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.812715    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-96ab73-29889450-lrpkk"]
Oct 30 13:30:20.871929 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.871839    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"]
Oct 30 13:30:20.875387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.875352    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-3e8ee4-29889450-vnkn2"]
Oct 30 13:30:20.908202 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.908170    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"]
Oct 30 13:30:20.911491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:20.911467    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-627dfa-29889450-6tm7r"]
Oct 30 13:30:21.601673 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:21.601624    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:30:21.602173 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:21.601700    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn" podUID="91c3dae9-4165-4660-bff5-f728025903cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 13:30:22.534903 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:22.534864    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2" path="/var/lib/kubelet/pods/871e8c5b-99ee-48f1-8a0a-ae1846253ec2/volumes"
Oct 30 13:30:22.535636 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:22.535614    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d60b5df-0998-4618-b201-4793314d793d" path="/var/lib/kubelet/pods/8d60b5df-0998-4618-b201-4793314d793d/volumes"
Oct 30 13:30:22.536102 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:22.536087    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9484142-6e89-4dc7-89ff-0fcd438cfedb" path="/var/lib/kubelet/pods/c9484142-6e89-4dc7-89ff-0fcd438cfedb/volumes"
Oct 30 13:30:22.601445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:22.601411    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:30:22.601646 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:22.601466    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn" podUID="91c3dae9-4165-4660-bff5-f728025903cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 13:30:23.602182 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:23.602144    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:30:23.602665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:23.602210    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn" podUID="91c3dae9-4165-4660-bff5-f728025903cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 13:30:24.583501 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:24.583470    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"
Oct 30 13:30:24.583676 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:24.583552    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"
Oct 30 13:30:24.601893 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:24.601860    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:30:24.602050 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:24.601926    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn" podUID="91c3dae9-4165-4660-bff5-f728025903cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 13:30:25.601757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:25.601721    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Oct 30 13:30:25.602258 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:25.601780    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn" podUID="91c3dae9-4165-4660-bff5-f728025903cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Oct 30 13:30:26.608124 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:26.608092    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:26.608638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:26.608619    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:26.609328 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:26.609308    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:30:32.861310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:32.861232    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"]
Oct 30 13:30:39.852979 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:39.852806    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"]
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.991345    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"]
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992140    5893 cpu_manager.go:414] "containerMap: removing container" podUID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992156    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992168    5893 cpu_manager.go:414] "containerMap: removing container" podUID="c9484142-6e89-4dc7-89ff-0fcd438cfedb" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992173    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9484142-6e89-4dc7-89ff-0fcd438cfedb" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992188    5893 cpu_manager.go:414] "containerMap: removing container" podUID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992193    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992201    5893 cpu_manager.go:414] "containerMap: removing container" podUID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992206    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992214    5893 cpu_manager.go:414] "containerMap: removing container" podUID="c9484142-6e89-4dc7-89ff-0fcd438cfedb" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992218    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9484142-6e89-4dc7-89ff-0fcd438cfedb" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992240    5893 cpu_manager.go:414] "containerMap: removing container" podUID="8d60b5df-0998-4618-b201-4793314d793d" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992244    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d60b5df-0998-4618-b201-4793314d793d" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992286    5893 cpu_manager.go:414] "containerMap: removing container" podUID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992293    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992305    5893 cpu_manager.go:414] "containerMap: removing container" podUID="8d60b5df-0998-4618-b201-4793314d793d" containerName="cleanup"
Oct 30 13:30:40.992298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992313    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d60b5df-0998-4618-b201-4793314d793d" containerName="cleanup"
Oct 30 13:30:40.993617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992323    5893 cpu_manager.go:414] "containerMap: removing container" podUID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a" containerName="cleanup"
Oct 30 13:30:40.993617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992328    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a" containerName="cleanup"
Oct 30 13:30:40.993617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992467    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9484142-6e89-4dc7-89ff-0fcd438cfedb" containerName="cleanup"
Oct 30 13:30:40.993617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992477    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d60b5df-0998-4618-b201-4793314d793d" containerName="cleanup"
Oct 30 13:30:40.993617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992485    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a" containerName="cleanup"
Oct 30 13:30:40.993617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992490    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2" containerName="cleanup"
Oct 30 13:30:40.993617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992496    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d60b5df-0998-4618-b201-4793314d793d" containerName="cleanup"
Oct 30 13:30:40.993617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992504    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a" containerName="cleanup"
Oct 30 13:30:40.993617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992513    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a" containerName="cleanup"
Oct 30 13:30:40.993617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992519    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2" containerName="cleanup"
Oct 30 13:30:40.993617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992524    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="871e8c5b-99ee-48f1-8a0a-ae1846253ec2" containerName="cleanup"
Oct 30 13:30:40.993617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.992531    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9484142-6e89-4dc7-89ff-0fcd438cfedb" containerName="cleanup"
Oct 30 13:30:40.997412 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:40.997390    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.035081 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.035057    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3c1b28-openshift-default-dockercfg-rbqkl\""
Oct 30 13:30:41.051389 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.051359    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"]
Oct 30 13:30:41.124054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.124021    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262ca860-a303-4568-935a-bb158b4dc3f3-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.124227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.124120    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262ca860-a303-4568-935a-bb158b4dc3f3-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.124227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.124180    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.124227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.124221    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.124358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.124307    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262ca860-a303-4568-935a-bb158b4dc3f3-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.124393 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.124374    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.124430 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.124395    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hdxl\" (UniqueName: \"kubernetes.io/projected/262ca860-a303-4568-935a-bb158b4dc3f3-kube-api-access-6hdxl\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.124499 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.124487    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.124572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.124561    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.226043 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.226012    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.226221 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.226108    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.226221 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.226171    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262ca860-a303-4568-935a-bb158b4dc3f3-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.226362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.226227    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262ca860-a303-4568-935a-bb158b4dc3f3-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.226362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.226295    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.226362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.226347    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.226512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.226391    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262ca860-a303-4568-935a-bb158b4dc3f3-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.226512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.226448    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.226512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.226477    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hdxl\" (UniqueName: \"kubernetes.io/projected/262ca860-a303-4568-935a-bb158b4dc3f3-kube-api-access-6hdxl\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.226669 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.226566    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.226726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.226666    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.226972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.226897    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262ca860-a303-4568-935a-bb158b4dc3f3-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.226972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.226923    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.227089 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.226978    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.228622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.228599    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.228934 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.228917    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262ca860-a303-4568-935a-bb158b4dc3f3-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.255077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.254974    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262ca860-a303-4568-935a-bb158b4dc3f3-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.256061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.256013    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hdxl\" (UniqueName: \"kubernetes.io/projected/262ca860-a303-4568-935a-bb158b4dc3f3-kube-api-access-6hdxl\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.330295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.320587    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:41.756560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.756529    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"]
Oct 30 13:30:41.759455 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:30:41.759426    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262ca860_a303_4568_935a_bb158b4dc3f3.slice/crio-d116fc362f7d85397518df0c3a5b54a51cdd6d1927e215d9b63213903520c491 WatchSource:0}: Error finding container d116fc362f7d85397518df0c3a5b54a51cdd6d1927e215d9b63213903520c491: Status 404 returned error can't find the container with id d116fc362f7d85397518df0c3a5b54a51cdd6d1927e215d9b63213903520c491
Oct 30 13:30:41.762816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.762781    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:41.762936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.762881    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:41.762936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:41.762925    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:42.033400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:42.033306    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" event={"ID":"262ca860-a303-4568-935a-bb158b4dc3f3","Type":"ContainerStarted","Data":"4cb1a2da5908989966a92e685ca67dcd5fac3da34a3a72e94d18d65796d786f8"}
Oct 30 13:30:42.033400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:42.033372    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" event={"ID":"262ca860-a303-4568-935a-bb158b4dc3f3","Type":"ContainerStarted","Data":"d116fc362f7d85397518df0c3a5b54a51cdd6d1927e215d9b63213903520c491"}
Oct 30 13:30:42.080889 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:42.080835    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" podStartSLOduration=2.080817673 podStartE2EDuration="2.080817673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:42.077896428 +0000 UTC m=+2683.661326122" watchObservedRunningTime="2026-10-30 13:30:42.080817673 +0000 UTC m=+2683.664247371"
Oct 30 13:30:42.321476 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:42.321398    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:42.327327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:42.327294    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:43.047297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:43.042752    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:43.070135 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:43.066668    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:30:47.898838 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:47.898800    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z" podUID="93a3e37c-f7cb-4cc4-8272-19293375dcac" containerName="main" containerID="cri-o://14b1f9b88a0b7f5d243b64ef45562e145bc37dbb8b41814ae5a4e0ad65275f77" gracePeriod=45
Oct 30 13:30:48.003524 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.003453    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z" podUID="93a3e37c-f7cb-4cc4-8272-19293375dcac" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": dial tcp 10.134.0.158:8000: connect: connection refused"
Oct 30 13:30:48.189526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.189418    5893 generic.go:356] "Generic (PLEG): container finished" podID="93a3e37c-f7cb-4cc4-8272-19293375dcac" containerID="14b1f9b88a0b7f5d243b64ef45562e145bc37dbb8b41814ae5a4e0ad65275f77" exitCode=0
Oct 30 13:30:48.189690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.189618    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z" event={"ID":"93a3e37c-f7cb-4cc4-8272-19293375dcac","Type":"ContainerDied","Data":"14b1f9b88a0b7f5d243b64ef45562e145bc37dbb8b41814ae5a4e0ad65275f77"}
Oct 30 13:30:48.250593 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.250568    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:30:48.369912 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.369867    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-home\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-home\") pod \"93a3e37c-f7cb-4cc4-8272-19293375dcac\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") "
Oct 30 13:30:48.370239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.370206    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-home" pod "93a3e37c-f7cb-4cc4-8272-19293375dcac" (UID: "93a3e37c-f7cb-4cc4-8272-19293375dcac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:48.370443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.370347    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a3e37c-f7cb-4cc4-8272-19293375dcac-tls-certs\" (UniqueName: \"kubernetes.io/secret/93a3e37c-f7cb-4cc4-8272-19293375dcac-tls-certs\") pod \"93a3e37c-f7cb-4cc4-8272-19293375dcac\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") "
Oct 30 13:30:48.370854 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.370813    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-tmp-dir\") pod \"93a3e37c-f7cb-4cc4-8272-19293375dcac\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") "
Oct 30 13:30:48.371057 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.371035    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-model-cache\") pod \"93a3e37c-f7cb-4cc4-8272-19293375dcac\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") "
Oct 30 13:30:48.371237 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.371217    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a3e37c-f7cb-4cc4-8272-19293375dcac-kube-api-access-w4484\" (UniqueName: \"kubernetes.io/projected/93a3e37c-f7cb-4cc4-8272-19293375dcac-kube-api-access-w4484\") pod \"93a3e37c-f7cb-4cc4-8272-19293375dcac\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") "
Oct 30 13:30:48.371458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.371439    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-dshm\") pod \"93a3e37c-f7cb-4cc4-8272-19293375dcac\" (UID: \"93a3e37c-f7cb-4cc4-8272-19293375dcac\") "
Oct 30 13:30:48.371629 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.371602    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-tmp-dir" pod "93a3e37c-f7cb-4cc4-8272-19293375dcac" (UID: "93a3e37c-f7cb-4cc4-8272-19293375dcac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:48.381009 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.372027    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-model-cache" pod "93a3e37c-f7cb-4cc4-8272-19293375dcac" (UID: "93a3e37c-f7cb-4cc4-8272-19293375dcac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:48.381009 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.379245    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-dshm" pod "93a3e37c-f7cb-4cc4-8272-19293375dcac" (UID: "93a3e37c-f7cb-4cc4-8272-19293375dcac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:48.390318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.390242    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a3e37c-f7cb-4cc4-8272-19293375dcac-tls-certs" pod "93a3e37c-f7cb-4cc4-8272-19293375dcac" (UID: "93a3e37c-f7cb-4cc4-8272-19293375dcac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:30:48.392211 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.391235    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a3e37c-f7cb-4cc4-8272-19293375dcac-kube-api-access-w4484" pod "93a3e37c-f7cb-4cc4-8272-19293375dcac" (UID: "93a3e37c-f7cb-4cc4-8272-19293375dcac"). InnerVolumeSpecName "kube-api-access-w4484". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:48.393665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.392978    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:48.393665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.393011    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93a3e37c-f7cb-4cc4-8272-19293375dcac-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:48.393665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.393028    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:48.393665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.393042    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:48.393665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.393058    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4484\" (UniqueName: \"kubernetes.io/projected/93a3e37c-f7cb-4cc4-8272-19293375dcac-kube-api-access-w4484\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:48.393665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:48.393074    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93a3e37c-f7cb-4cc4-8272-19293375dcac-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:48.649440 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:48.649402    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:30:49.202554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:49.202521    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"
Oct 30 13:30:49.203035 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:49.202522    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z" event={"ID":"93a3e37c-f7cb-4cc4-8272-19293375dcac","Type":"ContainerDied","Data":"1dec32d38ff8680eaeff02511c3bb1cf4508669851284757280b14b02f9b2f4a"}
Oct 30 13:30:49.203035 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:49.202670    5893 scope.go:122] "RemoveContainer" containerID="14b1f9b88a0b7f5d243b64ef45562e145bc37dbb8b41814ae5a4e0ad65275f77"
Oct 30 13:30:49.253740 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:49.253708    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"]
Oct 30 13:30:49.262690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:49.262662    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-bcd76c/sub-test-model-bcd76c-kserve-79f67fcc87-8ns5z"]
Oct 30 13:30:50.544580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:50.544543    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a3e37c-f7cb-4cc4-8272-19293375dcac" path="/var/lib/kubelet/pods/93a3e37c-f7cb-4cc4-8272-19293375dcac/volumes"
Oct 30 13:30:54.876120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:54.876074    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48" podUID="e67a9877-0091-4783-955c-bd4e98f420c5" containerName="main" containerID="cri-o://1129ef404e0b58d0a4185016da1029c10a3ebcd6ce044885eb8a3d5afce70eec" gracePeriod=45
Oct 30 13:30:55.204329 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.204290    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:30:55.222842 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.222797    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e67a9877-0091-4783-955c-bd4e98f420c5-tls-certs\" (UniqueName: \"kubernetes.io/secret/e67a9877-0091-4783-955c-bd4e98f420c5-tls-certs\") pod \"e67a9877-0091-4783-955c-bd4e98f420c5\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") "
Oct 30 13:30:55.223013 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.222879    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-dshm\") pod \"e67a9877-0091-4783-955c-bd4e98f420c5\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") "
Oct 30 13:30:55.223013 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.222915    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-tmp-dir\") pod \"e67a9877-0091-4783-955c-bd4e98f420c5\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") "
Oct 30 13:30:55.223122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.223024    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e67a9877-0091-4783-955c-bd4e98f420c5-kube-api-access-nj7gn\" (UniqueName: \"kubernetes.io/projected/e67a9877-0091-4783-955c-bd4e98f420c5-kube-api-access-nj7gn\") pod \"e67a9877-0091-4783-955c-bd4e98f420c5\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") "
Oct 30 13:30:55.223122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.223077    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-model-cache\") pod \"e67a9877-0091-4783-955c-bd4e98f420c5\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") "
Oct 30 13:30:55.223122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.223116    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-home\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-home\") pod \"e67a9877-0091-4783-955c-bd4e98f420c5\" (UID: \"e67a9877-0091-4783-955c-bd4e98f420c5\") "
Oct 30 13:30:55.223299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.223224    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-tmp-dir" pod "e67a9877-0091-4783-955c-bd4e98f420c5" (UID: "e67a9877-0091-4783-955c-bd4e98f420c5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:55.223530 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.223463    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-home" pod "e67a9877-0091-4783-955c-bd4e98f420c5" (UID: "e67a9877-0091-4783-955c-bd4e98f420c5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:55.223530 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.223484    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-model-cache" pod "e67a9877-0091-4783-955c-bd4e98f420c5" (UID: "e67a9877-0091-4783-955c-bd4e98f420c5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:55.223987 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.223970    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:55.224049 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.223989    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:55.224049 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.223998    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:55.225721 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.225693    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-dshm" pod "e67a9877-0091-4783-955c-bd4e98f420c5" (UID: "e67a9877-0091-4783-955c-bd4e98f420c5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:30:55.225838 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.225758    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e67a9877-0091-4783-955c-bd4e98f420c5-kube-api-access-nj7gn" pod "e67a9877-0091-4783-955c-bd4e98f420c5" (UID: "e67a9877-0091-4783-955c-bd4e98f420c5"). InnerVolumeSpecName "kube-api-access-nj7gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:55.226257 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.226237    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e67a9877-0091-4783-955c-bd4e98f420c5-tls-certs" pod "e67a9877-0091-4783-955c-bd4e98f420c5" (UID: "e67a9877-0091-4783-955c-bd4e98f420c5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:30:55.268406 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.268370    5893 generic.go:356] "Generic (PLEG): container finished" podID="e67a9877-0091-4783-955c-bd4e98f420c5" containerID="1129ef404e0b58d0a4185016da1029c10a3ebcd6ce044885eb8a3d5afce70eec" exitCode=0
Oct 30 13:30:55.268598 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.268457    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48" event={"ID":"e67a9877-0091-4783-955c-bd4e98f420c5","Type":"ContainerDied","Data":"1129ef404e0b58d0a4185016da1029c10a3ebcd6ce044885eb8a3d5afce70eec"}
Oct 30 13:30:55.268598 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.268494    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48" event={"ID":"e67a9877-0091-4783-955c-bd4e98f420c5","Type":"ContainerDied","Data":"bb5350f4ea95212d9e8f281b9b33fa95b1b52e7fb58607ee784466f6a1b63cee"}
Oct 30 13:30:55.268598 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.268516    5893 scope.go:122] "RemoveContainer" containerID="1129ef404e0b58d0a4185016da1029c10a3ebcd6ce044885eb8a3d5afce70eec"
Oct 30 13:30:55.268764 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.268658    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"
Oct 30 13:30:55.292067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.292043    5893 scope.go:122] "RemoveContainer" containerID="1129ef404e0b58d0a4185016da1029c10a3ebcd6ce044885eb8a3d5afce70eec"
Oct 30 13:30:55.292460 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:30:55.292438    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1129ef404e0b58d0a4185016da1029c10a3ebcd6ce044885eb8a3d5afce70eec\": container with ID starting with 1129ef404e0b58d0a4185016da1029c10a3ebcd6ce044885eb8a3d5afce70eec not found: ID does not exist" containerID="1129ef404e0b58d0a4185016da1029c10a3ebcd6ce044885eb8a3d5afce70eec"
Oct 30 13:30:55.292553 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.292471    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1129ef404e0b58d0a4185016da1029c10a3ebcd6ce044885eb8a3d5afce70eec"} err="failed to get container status \"1129ef404e0b58d0a4185016da1029c10a3ebcd6ce044885eb8a3d5afce70eec\": rpc error: code = NotFound desc = could not find container \"1129ef404e0b58d0a4185016da1029c10a3ebcd6ce044885eb8a3d5afce70eec\": container with ID starting with 1129ef404e0b58d0a4185016da1029c10a3ebcd6ce044885eb8a3d5afce70eec not found: ID does not exist"
Oct 30 13:30:55.325089 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.325058    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e67a9877-0091-4783-955c-bd4e98f420c5-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:55.325089 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.325086    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e67a9877-0091-4783-955c-bd4e98f420c5-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:55.325089 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.325096    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj7gn\" (UniqueName: \"kubernetes.io/projected/e67a9877-0091-4783-955c-bd4e98f420c5-kube-api-access-nj7gn\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:30:55.332135 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.332105    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"]
Oct 30 13:30:55.340106 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.340083    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-0e874e/sub-test-model-0e874e-kserve-786d99979f-vfq48"]
Oct 30 13:30:55.360993 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.360963    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"]
Oct 30 13:30:55.361783 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.361764    5893 cpu_manager.go:414] "containerMap: removing container" podUID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a" containerName="cleanup"
Oct 30 13:30:55.361783 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.361783    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78c9276f-5e11-4866-8a30-1bb4c38a5e8a" containerName="cleanup"
Oct 30 13:30:55.361949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.361804    5893 cpu_manager.go:414] "containerMap: removing container" podUID="c9484142-6e89-4dc7-89ff-0fcd438cfedb" containerName="cleanup"
Oct 30 13:30:55.361949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.361810    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9484142-6e89-4dc7-89ff-0fcd438cfedb" containerName="cleanup"
Oct 30 13:30:55.361949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.361846    5893 cpu_manager.go:414] "containerMap: removing container" podUID="e67a9877-0091-4783-955c-bd4e98f420c5" containerName="main"
Oct 30 13:30:55.361949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.361851    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e67a9877-0091-4783-955c-bd4e98f420c5" containerName="main"
Oct 30 13:30:55.361949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.361862    5893 cpu_manager.go:414] "containerMap: removing container" podUID="8d60b5df-0998-4618-b201-4793314d793d" containerName="cleanup"
Oct 30 13:30:55.361949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.361867    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d60b5df-0998-4618-b201-4793314d793d" containerName="cleanup"
Oct 30 13:30:55.361949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.361874    5893 cpu_manager.go:414] "containerMap: removing container" podUID="93a3e37c-f7cb-4cc4-8272-19293375dcac" containerName="main"
Oct 30 13:30:55.361949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.361878    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a3e37c-f7cb-4cc4-8272-19293375dcac" containerName="main"
Oct 30 13:30:55.362358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.361978    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9484142-6e89-4dc7-89ff-0fcd438cfedb" containerName="cleanup"
Oct 30 13:30:55.362358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.361988    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="e67a9877-0091-4783-955c-bd4e98f420c5" containerName="main"
Oct 30 13:30:55.362358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.361995    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d60b5df-0998-4618-b201-4793314d793d" containerName="cleanup"
Oct 30 13:30:55.362358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.362000    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a3e37c-f7cb-4cc4-8272-19293375dcac" containerName="main"
Oct 30 13:30:55.367312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.367263    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.390398 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.390363    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"]
Oct 30 13:30:55.426542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.426434    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.426542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.426502    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.426780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.426581    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e84f541-2caa-40dc-9b56-f320c72f5097-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.426780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.426693    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.426780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.426753    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.426951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.426871    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.426951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.426941    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.427063 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.427041    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.427206 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.427185    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5ngn\" (UniqueName: \"kubernetes.io/projected/1e84f541-2caa-40dc-9b56-f320c72f5097-kube-api-access-m5ngn\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.529178 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.529146    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.529358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.529189    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.529358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.529314    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e84f541-2caa-40dc-9b56-f320c72f5097-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.529606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.529577    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.529743 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.529626    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.529743 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.529721    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.529857 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.529792    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.529857 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.529841    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.529973 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.529940    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5ngn\" (UniqueName: \"kubernetes.io/projected/1e84f541-2caa-40dc-9b56-f320c72f5097-kube-api-access-m5ngn\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.530034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.529962    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-workload-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.530114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.530089    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e84f541-2caa-40dc-9b56-f320c72f5097-istiod-ca-cert\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.530196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.530175    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-workload-certs\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.530363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.530259    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-credential-socket\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.530556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.530396    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-data\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.531856 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.531837    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-podinfo\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.532096 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.532079    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-envoy\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.551408 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.551382    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-token\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.551882 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.551858    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5ngn\" (UniqueName: \"kubernetes.io/projected/1e84f541-2caa-40dc-9b56-f320c72f5097-kube-api-access-m5ngn\") pod \"e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") " pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.683565 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.683460    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:55.828170 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:55.828125    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"]
Oct 30 13:30:56.046797 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:56.046770    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"]
Oct 30 13:30:56.049485 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:30:56.049448    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e84f541_2caa_40dc_9b56_f320c72f5097.slice/crio-f59a8ee7d5e1591aeaaab932ceb890fd698facc0f21eb0d9c57d1f2a1e395f12 WatchSource:0}: Error finding container f59a8ee7d5e1591aeaaab932ceb890fd698facc0f21eb0d9c57d1f2a1e395f12: Status 404 returned error can't find the container with id f59a8ee7d5e1591aeaaab932ceb890fd698facc0f21eb0d9c57d1f2a1e395f12
Oct 30 13:30:56.052775 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:56.052741    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:56.052894 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:56.052829    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:56.052894 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:56.052878    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:30:56.309392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:56.302051    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz" event={"ID":"1e84f541-2caa-40dc-9b56-f320c72f5097","Type":"ContainerStarted","Data":"34e990fe417fe794072cafeb38ede2453d16da8d8f8b390fe9735a5adb817e00"}
Oct 30 13:30:56.309392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:56.302168    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz" event={"ID":"1e84f541-2caa-40dc-9b56-f320c72f5097","Type":"ContainerStarted","Data":"f59a8ee7d5e1591aeaaab932ceb890fd698facc0f21eb0d9c57d1f2a1e395f12"}
Oct 30 13:30:56.346824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:56.346763    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz" podStartSLOduration=1.346747933 podStartE2EDuration="1.346747933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:56.342468253 +0000 UTC m=+2697.925897946" watchObservedRunningTime="2026-10-30 13:30:56.346747933 +0000 UTC m=+2697.930177626"
Oct 30 13:30:56.536906 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:56.536863    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e67a9877-0091-4783-955c-bd4e98f420c5" path="/var/lib/kubelet/pods/e67a9877-0091-4783-955c-bd4e98f420c5/volumes"
Oct 30 13:30:56.684510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:56.684430    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:30:57.686396 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:57.686360    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 30 13:30:57.686926 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:57.686422    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz" podUID="1e84f541-2caa-40dc-9b56-f320c72f5097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 30 13:30:58.684548 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:58.684451    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 30 13:30:58.684548 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:58.684500    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz" podUID="1e84f541-2caa-40dc-9b56-f320c72f5097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 30 13:30:59.684852 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:59.684810    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 30 13:30:59.685315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:30:59.684891    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz" podUID="1e84f541-2caa-40dc-9b56-f320c72f5097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 30 13:31:00.684807 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:00.684772    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 30 13:31:00.685064 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:00.684839    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz" podUID="1e84f541-2caa-40dc-9b56-f320c72f5097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 30 13:31:00.773131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:00.773090    5893 scope.go:122] "RemoveContainer" containerID="255da8addcadf2572c0ad665014a5163d16ee9a523a99044e79362d90510fdb8"
Oct 30 13:31:00.829413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:00.829361    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb" podUID="e65e23b4-252a-4316-8b34-87fec8e13f8b" containerName="payload-processing" containerID="cri-o://46292febe9458a4f44965e49ef403da9efd077cc01cef92a3252a96c485239f9" gracePeriod=25
Oct 30 13:31:01.224053 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.224020    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:31:01.252299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.247121    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"
Oct 30 13:31:01.252299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.248716    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:31:01.313423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.313057    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:31:01.329377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.329352    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:31:01.337028 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.337000    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e65e23b4-252a-4316-8b34-87fec8e13f8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65e23b4-252a-4316-8b34-87fec8e13f8b-plugins-config-volume\") pod \"e65e23b4-252a-4316-8b34-87fec8e13f8b\" (UID: \"e65e23b4-252a-4316-8b34-87fec8e13f8b\") "
Oct 30 13:31:01.337170 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.337082    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65e23b4-252a-4316-8b34-87fec8e13f8b-kube-api-access-2qt9p\" (UniqueName: \"kubernetes.io/projected/e65e23b4-252a-4316-8b34-87fec8e13f8b-kube-api-access-2qt9p\") pod \"e65e23b4-252a-4316-8b34-87fec8e13f8b\" (UID: \"e65e23b4-252a-4316-8b34-87fec8e13f8b\") "
Oct 30 13:31:01.337427 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.337403    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65e23b4-252a-4316-8b34-87fec8e13f8b-plugins-config-volume" pod "e65e23b4-252a-4316-8b34-87fec8e13f8b" (UID: "e65e23b4-252a-4316-8b34-87fec8e13f8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:31:01.338046 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.338025    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e65e23b4-252a-4316-8b34-87fec8e13f8b-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:31:01.374432 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.340343    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65e23b4-252a-4316-8b34-87fec8e13f8b-kube-api-access-2qt9p" pod "e65e23b4-252a-4316-8b34-87fec8e13f8b" (UID: "e65e23b4-252a-4316-8b34-87fec8e13f8b"). InnerVolumeSpecName "kube-api-access-2qt9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:01.394357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.391260    5893 generic.go:356] "Generic (PLEG): container finished" podID="e65e23b4-252a-4316-8b34-87fec8e13f8b" containerID="46292febe9458a4f44965e49ef403da9efd077cc01cef92a3252a96c485239f9" exitCode=0
Oct 30 13:31:01.394357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.391310    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb" event={"ID":"e65e23b4-252a-4316-8b34-87fec8e13f8b","Type":"ContainerDied","Data":"46292febe9458a4f44965e49ef403da9efd077cc01cef92a3252a96c485239f9"}
Oct 30 13:31:01.394357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.391363    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"
Oct 30 13:31:01.394357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.391390    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb" event={"ID":"e65e23b4-252a-4316-8b34-87fec8e13f8b","Type":"ContainerDied","Data":"ad9441be97350104c06c0d343c882f11167d4c1e9e0c6aed299d6cefd748c86a"}
Oct 30 13:31:01.394357 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.391416    5893 scope.go:122] "RemoveContainer" containerID="46292febe9458a4f44965e49ef403da9efd077cc01cef92a3252a96c485239f9"
Oct 30 13:31:01.405116 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.405097    5893 scope.go:122] "RemoveContainer" containerID="46292febe9458a4f44965e49ef403da9efd077cc01cef92a3252a96c485239f9"
Oct 30 13:31:01.406008 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:31:01.405980    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46292febe9458a4f44965e49ef403da9efd077cc01cef92a3252a96c485239f9\": container with ID starting with 46292febe9458a4f44965e49ef403da9efd077cc01cef92a3252a96c485239f9 not found: ID does not exist" containerID="46292febe9458a4f44965e49ef403da9efd077cc01cef92a3252a96c485239f9"
Oct 30 13:31:01.406116 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.406019    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46292febe9458a4f44965e49ef403da9efd077cc01cef92a3252a96c485239f9"} err="failed to get container status \"46292febe9458a4f44965e49ef403da9efd077cc01cef92a3252a96c485239f9\": rpc error: code = NotFound desc = could not find container \"46292febe9458a4f44965e49ef403da9efd077cc01cef92a3252a96c485239f9\": container with ID starting with 46292febe9458a4f44965e49ef403da9efd077cc01cef92a3252a96c485239f9 not found: ID does not exist"
Oct 30 13:31:01.439748 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.439720    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qt9p\" (UniqueName: \"kubernetes.io/projected/e65e23b4-252a-4316-8b34-87fec8e13f8b-kube-api-access-2qt9p\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:31:01.482438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.479249    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"]
Oct 30 13:31:01.484590 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.484565    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76405c-6588477f54-s89nb"]
Oct 30 13:31:01.684291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.684167    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 30 13:31:01.684291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:01.684225    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz" podUID="1e84f541-2caa-40dc-9b56-f320c72f5097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 30 13:31:02.534787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:02.534756    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e65e23b4-252a-4316-8b34-87fec8e13f8b" path="/var/lib/kubelet/pods/e65e23b4-252a-4316-8b34-87fec8e13f8b/volumes"
Oct 30 13:31:02.684296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:02.684203    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Oct 30 13:31:02.684296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:02.684255    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz" podUID="1e84f541-2caa-40dc-9b56-f320c72f5097" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Oct 30 13:31:03.690041 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:03.690012    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:31:03.690575 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:03.690556    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:31:03.691396 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:03.691376    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:31:18.652514 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:31:18.652445    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:31:20.464959 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:20.464924    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"]
Oct 30 13:31:33.147414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.147375    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"]
Oct 30 13:31:33.148347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.148328    5893 cpu_manager.go:414] "containerMap: removing container" podUID="e65e23b4-252a-4316-8b34-87fec8e13f8b" containerName="payload-processing"
Oct 30 13:31:33.148406 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.148350    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65e23b4-252a-4316-8b34-87fec8e13f8b" containerName="payload-processing"
Oct 30 13:31:33.148516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.148506    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65e23b4-252a-4316-8b34-87fec8e13f8b" containerName="payload-processing"
Oct 30 13:31:33.152538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.152519    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:33.175216 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.175188    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3c1b28-serving-cert\""
Oct 30 13:31:33.176126 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.176096    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"]
Oct 30 13:31:33.222297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.222251    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24pxr\" (UniqueName: \"kubernetes.io/projected/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-kube-api-access-24pxr\") pod \"maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:33.222556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.222536    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-metrics-certs\") pod \"maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:33.222779 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.222752    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:33.222862 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.222801    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:33.323930 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.323903    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:33.324153 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.323938    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:33.324237 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.324157    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24pxr\" (UniqueName: \"kubernetes.io/projected/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-kube-api-access-24pxr\") pod \"maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:33.324237 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.324192    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-metrics-certs\") pod \"maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:33.324922 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.324893    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:33.356846 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.356263    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-metrics-certs\") pod \"maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:33.363685 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.357462    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24pxr\" (UniqueName: \"kubernetes.io/projected/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-kube-api-access-24pxr\") pod \"maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:33.370080 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.368847    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:33.477505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.477472    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:33.799788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.799763    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"]
Oct 30 13:31:33.802727 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:31:33.802693    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0bfdfdc_1f43_4127_b050_4e8a2c2f516f.slice/crio-d0988a180ec2596f291666616029f49ef77e8580884e4ff28617370b611f440a WatchSource:0}: Error finding container d0988a180ec2596f291666616029f49ef77e8580884e4ff28617370b611f440a: Status 404 returned error can't find the container with id d0988a180ec2596f291666616029f49ef77e8580884e4ff28617370b611f440a
Oct 30 13:31:33.804253 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:33.804234    5893 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:31:34.708534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:34.708439    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz" event={"ID":"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f","Type":"ContainerStarted","Data":"9ae9def9cd2a2b672780ae0fbf1d1f1317099ebf2adc4cca1b2ff900069eece4"}
Oct 30 13:31:34.708534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:34.708507    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:34.709000 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:34.708552    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz" event={"ID":"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f","Type":"ContainerStarted","Data":"d0988a180ec2596f291666616029f49ef77e8580884e4ff28617370b611f440a"}
Oct 30 13:31:34.768658 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:34.768603    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz" podStartSLOduration=1.354509224 podStartE2EDuration="1.768585417s" totalImagesPullingTime="414.076193ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:34.765175363 +0000 UTC m=+2736.348605057" watchObservedRunningTime="2026-10-30 13:31:34.768585417 +0000 UTC m=+2736.352015111"
Oct 30 13:31:35.492596 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.492557    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch" podUID="78dce1e2-01b4-42d2-bed5-7daeac16ffa9" containerName="main" containerID="cri-o://f5f251e0f16e6cc2d7705e03cbdee652681c41d2c51046223549870be2476678" gracePeriod=45
Oct 30 13:31:35.726809 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.726766    5893 generic.go:356] "Generic (PLEG): container finished" podID="78dce1e2-01b4-42d2-bed5-7daeac16ffa9" containerID="f5f251e0f16e6cc2d7705e03cbdee652681c41d2c51046223549870be2476678" exitCode=0
Oct 30 13:31:35.728004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.727972    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch" event={"ID":"78dce1e2-01b4-42d2-bed5-7daeac16ffa9","Type":"ContainerDied","Data":"f5f251e0f16e6cc2d7705e03cbdee652681c41d2c51046223549870be2476678"}
Oct 30 13:31:35.809515 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.809489    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:31:35.864018 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.863978    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-home\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-home\") pod \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") "
Oct 30 13:31:35.864225 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.864152    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-dshm\") pod \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") "
Oct 30 13:31:35.864323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.864242    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-model-cache\") pod \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") "
Oct 30 13:31:35.864440 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.864420    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-tls-certs\" (UniqueName: \"kubernetes.io/secret/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-tls-certs\") pod \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") "
Oct 30 13:31:35.864530 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.864485    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-tmp-dir\") pod \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") "
Oct 30 13:31:35.864657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.864637    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-kube-api-access-7r9xx\" (UniqueName: \"kubernetes.io/projected/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-kube-api-access-7r9xx\") pod \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\" (UID: \"78dce1e2-01b4-42d2-bed5-7daeac16ffa9\") "
Oct 30 13:31:35.864948 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.864922    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-home" pod "78dce1e2-01b4-42d2-bed5-7daeac16ffa9" (UID: "78dce1e2-01b4-42d2-bed5-7daeac16ffa9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:35.865578 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.865555    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:31:35.866670 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.866643    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-model-cache" pod "78dce1e2-01b4-42d2-bed5-7daeac16ffa9" (UID: "78dce1e2-01b4-42d2-bed5-7daeac16ffa9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:35.866834 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.866735    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-tmp-dir" pod "78dce1e2-01b4-42d2-bed5-7daeac16ffa9" (UID: "78dce1e2-01b4-42d2-bed5-7daeac16ffa9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:35.868915 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.868779    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-kube-api-access-7r9xx" pod "78dce1e2-01b4-42d2-bed5-7daeac16ffa9" (UID: "78dce1e2-01b4-42d2-bed5-7daeac16ffa9"). InnerVolumeSpecName "kube-api-access-7r9xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:35.869433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.869411    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-tls-certs" pod "78dce1e2-01b4-42d2-bed5-7daeac16ffa9" (UID: "78dce1e2-01b4-42d2-bed5-7daeac16ffa9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:31:35.870310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.870173    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-dshm" pod "78dce1e2-01b4-42d2-bed5-7daeac16ffa9" (UID: "78dce1e2-01b4-42d2-bed5-7daeac16ffa9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:35.967033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.967000    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:31:35.967033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.967025    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:31:35.967033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.967035    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:31:35.967376 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.967045    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:31:35.967376 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:35.967057    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r9xx\" (UniqueName: \"kubernetes.io/projected/78dce1e2-01b4-42d2-bed5-7daeac16ffa9-kube-api-access-7r9xx\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:31:36.738234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:36.738205    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"
Oct 30 13:31:36.738234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:36.738213    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch" event={"ID":"78dce1e2-01b4-42d2-bed5-7daeac16ffa9","Type":"ContainerDied","Data":"9670ac4968850d4823a0559e1a867e9f61e6f32e283bb7d605897c80aed4aaae"}
Oct 30 13:31:36.738782 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:36.738261    5893 scope.go:122] "RemoveContainer" containerID="f5f251e0f16e6cc2d7705e03cbdee652681c41d2c51046223549870be2476678"
Oct 30 13:31:36.834830 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:36.834748    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"]
Oct 30 13:31:36.844085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:36.844061    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-627dfa/auth-test-model-627dfa-kserve-86944d78b9-5cxch"]
Oct 30 13:31:38.533070 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:38.533033    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78dce1e2-01b4-42d2-bed5-7daeac16ffa9" path="/var/lib/kubelet/pods/78dce1e2-01b4-42d2-bed5-7daeac16ffa9/volumes"
Oct 30 13:31:40.734846 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:40.734814    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:31:41.278472 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.278431    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"]
Oct 30 13:31:41.284035 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.284001    5893 cpu_manager.go:414] "containerMap: removing container" podUID="78dce1e2-01b4-42d2-bed5-7daeac16ffa9" containerName="main"
Oct 30 13:31:41.284248 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.284235    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78dce1e2-01b4-42d2-bed5-7daeac16ffa9" containerName="main"
Oct 30 13:31:41.284631 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.284615    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="78dce1e2-01b4-42d2-bed5-7daeac16ffa9" containerName="main"
Oct 30 13:31:41.303544 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.303514    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"
Oct 30 13:31:41.317444 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.317415    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-76405c\""
Oct 30 13:31:41.318199 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.318172    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-76405c-dockercfg-xpqnk\""
Oct 30 13:31:41.335293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.330785    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"]
Oct 30 13:31:41.340040 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.339920    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020e8b33-8eb0-4a5e-9de7-28b422ef2b62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df\" (UID: \"020e8b33-8eb0-4a5e-9de7-28b422ef2b62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"
Oct 30 13:31:41.340040 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.340007    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxc8v\" (UniqueName: \"kubernetes.io/projected/020e8b33-8eb0-4a5e-9de7-28b422ef2b62-kube-api-access-gxc8v\") pod \"payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df\" (UID: \"020e8b33-8eb0-4a5e-9de7-28b422ef2b62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"
Oct 30 13:31:41.441213 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.441177    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020e8b33-8eb0-4a5e-9de7-28b422ef2b62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df\" (UID: \"020e8b33-8eb0-4a5e-9de7-28b422ef2b62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"
Oct 30 13:31:41.441402 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.441294    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxc8v\" (UniqueName: \"kubernetes.io/projected/020e8b33-8eb0-4a5e-9de7-28b422ef2b62-kube-api-access-gxc8v\") pod \"payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df\" (UID: \"020e8b33-8eb0-4a5e-9de7-28b422ef2b62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"
Oct 30 13:31:41.441965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.441945    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020e8b33-8eb0-4a5e-9de7-28b422ef2b62-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df\" (UID: \"020e8b33-8eb0-4a5e-9de7-28b422ef2b62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"
Oct 30 13:31:41.463318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.463260    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxc8v\" (UniqueName: \"kubernetes.io/projected/020e8b33-8eb0-4a5e-9de7-28b422ef2b62-kube-api-access-gxc8v\") pod \"payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df\" (UID: \"020e8b33-8eb0-4a5e-9de7-28b422ef2b62\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"
Oct 30 13:31:41.692201 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:41.692111    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"
Oct 30 13:31:42.013634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:42.013601    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"]
Oct 30 13:31:42.017823 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:31:42.017785    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod020e8b33_8eb0_4a5e_9de7_28b422ef2b62.slice/crio-6e72cada980e18d413061cf36237bc47500dc0ce834a2d72ceee9eaeff1227f0 WatchSource:0}: Error finding container 6e72cada980e18d413061cf36237bc47500dc0ce834a2d72ceee9eaeff1227f0: Status 404 returned error can't find the container with id 6e72cada980e18d413061cf36237bc47500dc0ce834a2d72ceee9eaeff1227f0
Oct 30 13:31:42.936899 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:42.936347    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df" event={"ID":"020e8b33-8eb0-4a5e-9de7-28b422ef2b62","Type":"ContainerStarted","Data":"6e72cada980e18d413061cf36237bc47500dc0ce834a2d72ceee9eaeff1227f0"}
Oct 30 13:31:44.957661 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:44.957622    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df" event={"ID":"020e8b33-8eb0-4a5e-9de7-28b422ef2b62","Type":"ContainerStarted","Data":"9bfe2088090375d09fbecb7057aedd7f731a99e488df6e11e8793ba06d1edc82"}
Oct 30 13:31:44.958107 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:44.957708    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"
Oct 30 13:31:45.001584 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:45.001531    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df" podStartSLOduration=2.075171619 podStartE2EDuration="4.0015146s" totalImagesPullingTime="1.926342981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:44.997263181 +0000 UTC m=+2746.580692886" watchObservedRunningTime="2026-10-30 13:31:45.0015146 +0000 UTC m=+2746.584944303"
Oct 30 13:31:48.693947 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:31:48.693907    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:31:56.008296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:56.008079    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"
Oct 30 13:31:59.717862 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:59.717829    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"]
Oct 30 13:31:59.723305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:59.723288    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:31:59.764827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:59.764798    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1c8130-openshift-default-dockercfg-drhsq\""
Oct 30 13:31:59.982786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:59.982679    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:31:59.994542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:59.994502    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:31:59.994957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:59.994926    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:31:59.995172 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:59.995151    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:31:59.995389 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:59.995370    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:31:59.995562 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:59.995543    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:31:59.995699 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:59.995682    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68wn\" (UniqueName: \"kubernetes.io/projected/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-kube-api-access-z68wn\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:31:59.995836 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:59.995820    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:31:59.995995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:31:59.995972    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.102859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.102809    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.103167 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.103145    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.103409 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.103389    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.103648 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.103633    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.103780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.103766    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68wn\" (UniqueName: \"kubernetes.io/projected/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-kube-api-access-z68wn\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.103916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.103902    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.104040 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.104027    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.104377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.104360    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.104692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.104673    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.139297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.108008    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.150954 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.124942    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.151219 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.126683    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.267246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.126914    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.267506 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.127966    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.267624 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.156358    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"]
Oct 30 13:32:00.267775 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.171835    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.267921 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.191874    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.268030 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.194425    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.268129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.196252    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68wn\" (UniqueName: \"kubernetes.io/projected/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-kube-api-access-z68wn\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.403312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.393649    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:00.914063 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.914034    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"]
Oct 30 13:32:00.915746 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:32:00.915719    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1530d7_d75e_4d05_9fda_46f3a07b60c1.slice/crio-470f973c32bff4c74b3a652cb38cf741e7decc47dcd61223c9b0a16022c41e24 WatchSource:0}: Error finding container 470f973c32bff4c74b3a652cb38cf741e7decc47dcd61223c9b0a16022c41e24: Status 404 returned error can't find the container with id 470f973c32bff4c74b3a652cb38cf741e7decc47dcd61223c9b0a16022c41e24
Oct 30 13:32:00.920137 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.920102    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:00.920262 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.920190    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:00.920262 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:00.920238    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:01.316315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:01.316075    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" event={"ID":"8b1530d7-d75e-4d05-9fda-46f3a07b60c1","Type":"ContainerStarted","Data":"6760e5c07c8e77766167a484dccf47838318d578fd146c9395330692f1de6431"}
Oct 30 13:32:01.316315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:01.316129    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" event={"ID":"8b1530d7-d75e-4d05-9fda-46f3a07b60c1","Type":"ContainerStarted","Data":"470f973c32bff4c74b3a652cb38cf741e7decc47dcd61223c9b0a16022c41e24"}
Oct 30 13:32:02.394301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:02.393450    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" podStartSLOduration=3.39342763 podStartE2EDuration="3.39342763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:02.377793387 +0000 UTC m=+2763.961223109" watchObservedRunningTime="2026-10-30 13:32:02.39342763 +0000 UTC m=+2763.976857323"
Oct 30 13:32:02.409470 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:02.409428    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:02.415912 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:02.415887    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:03.332823 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:03.332795    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:03.333640 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:03.333618    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:32:11.244915 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:11.244621    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:32:11.245619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:11.244990    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:11.570314 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:11.570213    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"]
Oct 30 13:32:11.935336 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:11.935213    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"]
Oct 30 13:32:11.935518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:11.935364    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.035653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.035625    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.035802 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.035793    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.035855 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.035838    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjpx5\" (UniqueName: \"kubernetes.io/projected/5e392435-687c-4ead-b5c2-44d275dd61d5-kube-api-access-xjpx5\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.035960 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.035939    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.036099 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.036077    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.036240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.036225    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.036346 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.036257    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.036346 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.036318    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.036425 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.036407    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e392435-687c-4ead-b5c2-44d275dd61d5-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.138382 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.138331    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.138382 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.138381    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.138687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.138413    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.138687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.138462    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e392435-687c-4ead-b5c2-44d275dd61d5-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.138687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.138506    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.138687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.138615    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.138687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.138676    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjpx5\" (UniqueName: \"kubernetes.io/projected/5e392435-687c-4ead-b5c2-44d275dd61d5-kube-api-access-xjpx5\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.138960 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.138744    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.138960 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.138884    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.139740 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.139406    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.139740 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.139678    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.139740 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.139679    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e392435-687c-4ead-b5c2-44d275dd61d5-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.139981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.139777    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.139981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.139923    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.523711 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.523679    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.524258 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.523720    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.524428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.523857    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjpx5\" (UniqueName: \"kubernetes.io/projected/5e392435-687c-4ead-b5c2-44d275dd61d5-kube-api-access-xjpx5\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.524428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.523910    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:12.554463 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:12.554437    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:14.083461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:14.083420    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:32:14.083980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:14.083496    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:16.483317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:16.479459    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"]
Oct 30 13:32:16.490620 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:16.490255    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:16.490620 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:16.490420    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:16.490620 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:16.490514    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:16.955548 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:16.955457    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs" event={"ID":"5e392435-687c-4ead-b5c2-44d275dd61d5","Type":"ContainerStarted","Data":"2891664e4348d22eeac18f21e22de7412af541e7c486743763bb12331509628c"}
Oct 30 13:32:16.955548 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:16.955518    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs" event={"ID":"5e392435-687c-4ead-b5c2-44d275dd61d5","Type":"ContainerStarted","Data":"ad7288eca0adcda8e6bc9a4c7fab484b4b029a822bb1dd14df76609937eb3bdb"}
Oct 30 13:32:17.029994 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:17.029922    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs" podStartSLOduration=6.02990181 podStartE2EDuration="6.02990181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:17.01542721 +0000 UTC m=+2778.598856920" watchObservedRunningTime="2026-10-30 13:32:17.02990181 +0000 UTC m=+2778.613331503"
Oct 30 13:32:17.571383 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:17.571340    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:17.572042 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:17.571876    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:32:17.572042 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:17.571938    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs" podUID="5e392435-687c-4ead-b5c2-44d275dd61d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 13:32:18.562419 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:18.562386    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:32:18.563422 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:18.563386    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs" podUID="5e392435-687c-4ead-b5c2-44d275dd61d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 13:32:18.994587 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:32:18.994551    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:32:19.564299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:19.558607    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:32:19.564299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:19.558666    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs" podUID="5e392435-687c-4ead-b5c2-44d275dd61d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 13:32:20.556785 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:20.556750    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:32:20.557414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:20.557382    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs" podUID="5e392435-687c-4ead-b5c2-44d275dd61d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 13:32:21.564184 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.562073    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:32:21.564184 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.562133    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs" podUID="5e392435-687c-4ead-b5c2-44d275dd61d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 13:32:21.629359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.629325    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"]
Oct 30 13:32:21.652308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.645700    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.693898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.693861    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"]
Oct 30 13:32:21.799399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.796515    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.799399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.796583    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhkh4\" (UniqueName: \"kubernetes.io/projected/5c56fcec-9f01-4323-a89f-ace02d7ac57a-kube-api-access-dhkh4\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.799399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.796650    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.799399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.796698    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.799399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.797032    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.799399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.797084    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.799399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.797210    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.799399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.797303    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.799399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.797332    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.898478    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.898543    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.898567    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.898622    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.898665    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhkh4\" (UniqueName: \"kubernetes.io/projected/5c56fcec-9f01-4323-a89f-ace02d7ac57a-kube-api-access-dhkh4\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.898727    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.898773    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.898870    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.898955    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.899028    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.899024    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.899700    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.899875    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.899928    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.902728    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.903837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.903056    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.920674 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.920648    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:21.922182 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:21.922159    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhkh4\" (UniqueName: \"kubernetes.io/projected/5c56fcec-9f01-4323-a89f-ace02d7ac57a-kube-api-access-dhkh4\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:22.018465 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:22.018364    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:22.472560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:22.459890    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"]
Oct 30 13:32:22.500548 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:32:22.500514    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c56fcec_9f01_4323_a89f_ace02d7ac57a.slice/crio-eecf68c74f20379eeabf0efb49cf7f8e6714c9a5ac9c6834c3090fffe2ab56ca WatchSource:0}: Error finding container eecf68c74f20379eeabf0efb49cf7f8e6714c9a5ac9c6834c3090fffe2ab56ca: Status 404 returned error can't find the container with id eecf68c74f20379eeabf0efb49cf7f8e6714c9a5ac9c6834c3090fffe2ab56ca
Oct 30 13:32:22.504790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:22.504750    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:22.504905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:22.504856    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:22.504970 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:22.504910    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:22.557412 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:22.555681    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Oct 30 13:32:22.557412 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:22.555741    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs" podUID="5e392435-687c-4ead-b5c2-44d275dd61d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Oct 30 13:32:23.077580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:23.077543    5893 generic.go:356] "Generic (PLEG): container finished" podID="262ca860-a303-4568-935a-bb158b4dc3f3" containerID="4cb1a2da5908989966a92e685ca67dcd5fac3da34a3a72e94d18d65796d786f8" exitCode=0
Oct 30 13:32:23.078669 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:23.077797    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" event={"ID":"262ca860-a303-4568-935a-bb158b4dc3f3","Type":"ContainerDied","Data":"4cb1a2da5908989966a92e685ca67dcd5fac3da34a3a72e94d18d65796d786f8"}
Oct 30 13:32:23.078669 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:23.078355    5893 scope.go:122] "RemoveContainer" containerID="4cb1a2da5908989966a92e685ca67dcd5fac3da34a3a72e94d18d65796d786f8"
Oct 30 13:32:23.080831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:23.080788    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:23.080951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:23.080867    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:23.080951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:23.080900    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:23.084627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:23.083497    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9" event={"ID":"5c56fcec-9f01-4323-a89f-ace02d7ac57a","Type":"ContainerStarted","Data":"01faf263e4099346606082156fb242751b5d5889f66665624bd720d1f576e1c1"}
Oct 30 13:32:23.084627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:23.083544    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9" event={"ID":"5c56fcec-9f01-4323-a89f-ace02d7ac57a","Type":"ContainerStarted","Data":"eecf68c74f20379eeabf0efb49cf7f8e6714c9a5ac9c6834c3090fffe2ab56ca"}
Oct 30 13:32:23.193507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:23.192114    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9" podStartSLOduration=2.192092046 podStartE2EDuration="2.192092046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:23.187374709 +0000 UTC m=+2784.770804419" watchObservedRunningTime="2026-10-30 13:32:23.192092046 +0000 UTC m=+2784.775521742"
Oct 30 13:32:23.573298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:23.571690    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:23.573515 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:23.573490    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:23.582041 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:23.581255    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:32:24.019038 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:24.018978    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:24.020815 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:24.020779    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:32:24.020967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:24.020846    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9" podUID="5c56fcec-9f01-4323-a89f-ace02d7ac57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 13:32:24.106505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:24.106468    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" event={"ID":"262ca860-a303-4568-935a-bb158b4dc3f3","Type":"ContainerStarted","Data":"cc89889bf848e635745624ed2b9657ef2f6b3f3cdc9263eb66e96099548f90d7"}
Oct 30 13:32:24.321469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:24.321390    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:32:24.322068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:24.322042    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:32:24.322165 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:24.322095    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 13:32:25.019044 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:25.019012    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:32:25.019320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:25.019061    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9" podUID="5c56fcec-9f01-4323-a89f-ace02d7ac57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 13:32:25.331350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:25.326140    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:32:25.331350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:25.326199    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 13:32:25.569295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:25.561713    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:32:25.569295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:25.561838    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:26.019841 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:26.019299    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:32:26.019841 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:26.019358    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9" podUID="5c56fcec-9f01-4323-a89f-ace02d7ac57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 13:32:26.131736 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:26.131648    5893 generic.go:356] "Generic (PLEG): container finished" podID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerID="c26c2dff88c0c911ef788fb28fb3c7c1be58d6589d6f694aa062aeaabbc6ddab" exitCode=0
Oct 30 13:32:26.131736 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:26.131719    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" event={"ID":"84c68bf3-ee6a-498f-aeb1-0318f788488d","Type":"ContainerDied","Data":"c26c2dff88c0c911ef788fb28fb3c7c1be58d6589d6f694aa062aeaabbc6ddab"}
Oct 30 13:32:26.132242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:26.132227    5893 scope.go:122] "RemoveContainer" containerID="c26c2dff88c0c911ef788fb28fb3c7c1be58d6589d6f694aa062aeaabbc6ddab"
Oct 30 13:32:26.134399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:26.134262    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:26.134494 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:26.134445    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:26.134494 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:26.134487    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:26.325124 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:26.324572    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:32:26.325124 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:26.324629    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 13:32:27.020044 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:27.020012    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:32:27.020578 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:27.020067    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9" podUID="5c56fcec-9f01-4323-a89f-ace02d7ac57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 13:32:27.216421 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:27.216383    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" event={"ID":"84c68bf3-ee6a-498f-aeb1-0318f788488d","Type":"ContainerStarted","Data":"8162d29a76e8454bb10d43803a71764ab59b229649e03d8731b13b1ba206cab8"}
Oct 30 13:32:27.324210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:27.322447    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:32:27.324210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:27.322493    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 13:32:27.802295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:27.801653    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:32:27.813770 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:27.808675    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:32:27.813770 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:27.808756    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 13:32:28.023509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:28.023479    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:32:28.024101 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:28.024069    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9" podUID="5c56fcec-9f01-4323-a89f-ace02d7ac57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 13:32:28.042103 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:28.042075    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:32:28.381004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:28.372929    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:32:28.381004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:28.372989    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 13:32:28.806177 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:28.800207    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:32:28.806177 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:28.800284    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 13:32:28.988301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:28.969383    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:29.019725 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:29.019686    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:32:29.020020 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:29.019993    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9" podUID="5c56fcec-9f01-4323-a89f-ace02d7ac57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 13:32:29.130511 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:29.130417    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:29.331449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:29.331405    5893 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:29.389355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:29.380785    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:32:29.389355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:29.380851    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 13:32:29.687351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:29.687014    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:29.735071 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:29.735024    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:32:29.826365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:29.823720    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:32:29.826365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:29.823804    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 13:32:30.020286 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.020232    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body=
Oct 30 13:32:30.034706 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.034668    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9" podUID="5c56fcec-9f01-4323-a89f-ace02d7ac57a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused"
Oct 30 13:32:30.137294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.132033    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"]
Oct 30 13:32:30.254452 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.239724    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"
Oct 30 13:32:30.267210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.255730    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a4f385\""
Oct 30 13:32:30.293347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.255988    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"]
Oct 30 13:32:30.293347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.269878    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"]
Oct 30 13:32:30.293347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.280343    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a4f385-dockercfg-66rcb\""
Oct 30 13:32:30.293347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.287471    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"]
Oct 30 13:32:30.293347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.287586    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"
Oct 30 13:32:30.379463 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.379420    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused" start-of-body=
Oct 30 13:32:30.379791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.379745    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.180:15021/healthz/ready\": dial tcp 10.134.0.180:15021: connect: connection refused"
Oct 30 13:32:30.413294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.408945    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c056ad-c2a2-4737-b1a6-4808325bb745-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p\" (UID: \"31c056ad-c2a2-4737-b1a6-4808325bb745\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"
Oct 30 13:32:30.430638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.430581    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7bcb\" (UniqueName: \"kubernetes.io/projected/31c056ad-c2a2-4737-b1a6-4808325bb745-kube-api-access-t7bcb\") pod \"payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p\" (UID: \"31c056ad-c2a2-4737-b1a6-4808325bb745\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"
Oct 30 13:32:30.430854 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.430669    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7716f549-910c-434d-88f9-a36ab4788e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5\" (UID: \"7716f549-910c-434d-88f9-a36ab4788e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"
Oct 30 13:32:30.430854 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.430808    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-796p7\" (UniqueName: \"kubernetes.io/projected/7716f549-910c-434d-88f9-a36ab4788e3f-kube-api-access-796p7\") pod \"payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5\" (UID: \"7716f549-910c-434d-88f9-a36ab4788e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"
Oct 30 13:32:30.533069 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.533035    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7bcb\" (UniqueName: \"kubernetes.io/projected/31c056ad-c2a2-4737-b1a6-4808325bb745-kube-api-access-t7bcb\") pod \"payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p\" (UID: \"31c056ad-c2a2-4737-b1a6-4808325bb745\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"
Oct 30 13:32:30.533287 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.533116    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7716f549-910c-434d-88f9-a36ab4788e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5\" (UID: \"7716f549-910c-434d-88f9-a36ab4788e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"
Oct 30 13:32:30.533287 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.533231    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-796p7\" (UniqueName: \"kubernetes.io/projected/7716f549-910c-434d-88f9-a36ab4788e3f-kube-api-access-796p7\") pod \"payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5\" (UID: \"7716f549-910c-434d-88f9-a36ab4788e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"
Oct 30 13:32:30.533430 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.533413    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c056ad-c2a2-4737-b1a6-4808325bb745-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p\" (UID: \"31c056ad-c2a2-4737-b1a6-4808325bb745\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"
Oct 30 13:32:30.533910 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.533882    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7716f549-910c-434d-88f9-a36ab4788e3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5\" (UID: \"7716f549-910c-434d-88f9-a36ab4788e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"
Oct 30 13:32:30.534072 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.534053    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c056ad-c2a2-4737-b1a6-4808325bb745-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p\" (UID: \"31c056ad-c2a2-4737-b1a6-4808325bb745\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"
Oct 30 13:32:30.573043 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.572965    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-796p7\" (UniqueName: \"kubernetes.io/projected/7716f549-910c-434d-88f9-a36ab4788e3f-kube-api-access-796p7\") pod \"payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5\" (UID: \"7716f549-910c-434d-88f9-a36ab4788e3f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"
Oct 30 13:32:30.599911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.599462    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7bcb\" (UniqueName: \"kubernetes.io/projected/31c056ad-c2a2-4737-b1a6-4808325bb745-kube-api-access-t7bcb\") pod \"payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p\" (UID: \"31c056ad-c2a2-4737-b1a6-4808325bb745\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"
Oct 30 13:32:30.733222 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.733161    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"
Oct 30 13:32:30.811504 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.811199    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"
Oct 30 13:32:30.811849 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.811819    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:32:30.811971 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:30.811870    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 13:32:31.036816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:31.036782    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:31.038680 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:31.038635    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:31.040160 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:31.040138    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:32:31.395168 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:31.394613    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"]
Oct 30 13:32:31.398460 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:32:31.398003    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7716f549_910c_434d_88f9_a36ab4788e3f.slice/crio-61c3b20b9370bad9d69299de54f98b6cc146f01130b2c36cc36d5a5d5a6dbf16 WatchSource:0}: Error finding container 61c3b20b9370bad9d69299de54f98b6cc146f01130b2c36cc36d5a5d5a6dbf16: Status 404 returned error can't find the container with id 61c3b20b9370bad9d69299de54f98b6cc146f01130b2c36cc36d5a5d5a6dbf16
Oct 30 13:32:31.402331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:31.402305    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:32:31.406244 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:31.406194    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:32:31.484353 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:32:31.483102    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c056ad_c2a2_4737_b1a6_4808325bb745.slice/crio-340de6670b1488b3e5c9638d5d4c1aa476147fd351487963b0a542bc72c271bf WatchSource:0}: Error finding container 340de6670b1488b3e5c9638d5d4c1aa476147fd351487963b0a542bc72c271bf: Status 404 returned error can't find the container with id 340de6670b1488b3e5c9638d5d4c1aa476147fd351487963b0a542bc72c271bf
Oct 30 13:32:31.493329 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:31.493302    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"]
Oct 30 13:32:31.798051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:31.797919    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:32:31.798051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:31.797974    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 13:32:32.284293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:32.284118    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5" event={"ID":"7716f549-910c-434d-88f9-a36ab4788e3f","Type":"ContainerStarted","Data":"92da90ff4fe1b9e1410e1c3f1b4f4c82132009a7ec3adacbfebbb0e9cc71639f"}
Oct 30 13:32:32.284293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:32.284180    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5" event={"ID":"7716f549-910c-434d-88f9-a36ab4788e3f","Type":"ContainerStarted","Data":"61c3b20b9370bad9d69299de54f98b6cc146f01130b2c36cc36d5a5d5a6dbf16"}
Oct 30 13:32:32.284293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:32.284211    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"
Oct 30 13:32:32.290831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:32.288302    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p" event={"ID":"31c056ad-c2a2-4737-b1a6-4808325bb745","Type":"ContainerStarted","Data":"8a1e273ad4a6320470957142c5e501682fbee0054dbbfebce1edad8508cd3593"}
Oct 30 13:32:32.290831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:32.288350    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"
Oct 30 13:32:32.290831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:32.288372    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p" event={"ID":"31c056ad-c2a2-4737-b1a6-4808325bb745","Type":"ContainerStarted","Data":"340de6670b1488b3e5c9638d5d4c1aa476147fd351487963b0a542bc72c271bf"}
Oct 30 13:32:32.333479 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:32.333414    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5" podStartSLOduration=3.333395855 podStartE2EDuration="3.333395855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:32.32962014 +0000 UTC m=+2793.913049836" watchObservedRunningTime="2026-10-30 13:32:32.333395855 +0000 UTC m=+2793.916825570"
Oct 30 13:32:32.377297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:32.377041    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p" podStartSLOduration=3.3770217000000002 podStartE2EDuration="3.3770217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:32.375369691 +0000 UTC m=+2793.958799407" watchObservedRunningTime="2026-10-30 13:32:32.3770217 +0000 UTC m=+2793.960451393"
Oct 30 13:32:32.797856 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:32.797782    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:32:32.798452 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:32.797859    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 13:32:33.798617 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:33.798581    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:32:33.799156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:33.798645    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Oct 30 13:32:34.803174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:34.803143    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:32:34.803694 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:34.803674    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:32:34.804474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:34.804455    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:32:43.299661 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:43.299574    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"
Oct 30 13:32:43.300372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:43.299667    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"
Oct 30 13:32:48.648874 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:32:48.648835    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:32:56.579186 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.579152    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"]
Oct 30 13:32:56.583698 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.583672    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.599462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.599066    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"]
Oct 30 13:32:56.641930 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.641897    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.642125 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.641945    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.642182 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.642111    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.642342 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.642322    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.642443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.642426    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsm8t\" (UniqueName: \"kubernetes.io/projected/30fc4668-2aed-4383-b5b1-092a5f3411a5-kube-api-access-vsm8t\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.642603 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.642587    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.642646 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.642630    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30fc4668-2aed-4383-b5b1-092a5f3411a5-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.642726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.642711    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.642816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.642796    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.744090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.744051    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.744319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.744154    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.744319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.744189    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.744319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.744220    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.744319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.744304    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.744526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.744435    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.744526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.744473    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-credential-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.744611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.744547    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsm8t\" (UniqueName: \"kubernetes.io/projected/30fc4668-2aed-4383-b5b1-092a5f3411a5-kube-api-access-vsm8t\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.744611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.744556    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-workload-certs\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.744739 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.744706    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.744824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.744753    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30fc4668-2aed-4383-b5b1-092a5f3411a5-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.744824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.744786    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-data\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.745031 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.744998    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-workload-socket\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.745505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.745433    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30fc4668-2aed-4383-b5b1-092a5f3411a5-istiod-ca-cert\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.747512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.747485    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-envoy\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.747677 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.747643    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-podinfo\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.765608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.765586    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-token\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.766091 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.766066    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsm8t\" (UniqueName: \"kubernetes.io/projected/30fc4668-2aed-4383-b5b1-092a5f3411a5-kube-api-access-vsm8t\") pod \"e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") " pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:56.906249 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:56.906166    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:57.245679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.245656    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"]
Oct 30 13:32:57.248919 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:32:57.248893    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fc4668_2aed_4383_b5b1_092a5f3411a5.slice/crio-15e2275736e633c56384dac49362f2f3c758865a38ba375b349c1ee3f1d4e353 WatchSource:0}: Error finding container 15e2275736e633c56384dac49362f2f3c758865a38ba375b349c1ee3f1d4e353: Status 404 returned error can't find the container with id 15e2275736e633c56384dac49362f2f3c758865a38ba375b349c1ee3f1d4e353
Oct 30 13:32:57.252555 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.252520    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:57.252653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.252588    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:57.252653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.252615    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:32:57.621652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.621540    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj" event={"ID":"30fc4668-2aed-4383-b5b1-092a5f3411a5","Type":"ContainerStarted","Data":"b39228adeb9377ff748803c785eff1b614622a8d92da47526c198440ead4224c"}
Oct 30 13:32:57.621652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.621601    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj" event={"ID":"30fc4668-2aed-4383-b5b1-092a5f3411a5","Type":"ContainerStarted","Data":"15e2275736e633c56384dac49362f2f3c758865a38ba375b349c1ee3f1d4e353"}
Oct 30 13:32:57.678519 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.678464    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj" podStartSLOduration=1.678449386 podStartE2EDuration="1.678449386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:57.67339957 +0000 UTC m=+2819.256829263" watchObservedRunningTime="2026-10-30 13:32:57.678449386 +0000 UTC m=+2819.261879118"
Oct 30 13:32:57.837500 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.837466    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"]
Oct 30 13:32:57.841468 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.841440    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:57.853858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.853825    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1c8130-serving-cert\""
Oct 30 13:32:57.864005 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.863975    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"]
Oct 30 13:32:57.906586 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.906501    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:32:57.967195 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.967164    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:57.967401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.967369    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-metrics-certs\") pod \"maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:57.967518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.967498    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kss59\" (UniqueName: \"kubernetes.io/projected/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-kube-api-access-kss59\") pod \"maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:57.967668 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:57.967636    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:58.071857 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:58.071817    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:58.072542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:58.072505    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-metrics-certs\") pod \"maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:58.072857 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:58.072823    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kss59\" (UniqueName: \"kubernetes.io/projected/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-kube-api-access-kss59\") pod \"maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:58.073120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:58.073085    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:58.076358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:58.076334    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:58.094518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:58.094484    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-metrics-certs\") pod \"maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:58.098945 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:58.098911    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:58.107547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:58.107513    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kss59\" (UniqueName: \"kubernetes.io/projected/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-kube-api-access-kss59\") pod \"maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:58.157471 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:58.157376    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:58.527758 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:32:58.527728    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa3efb6_6744_4a5e_8d30_3c1f23d79b34.slice/crio-bccac22a4af094e7f341de9271910388d0bafe43225a904229ad1bc9cf2322bc WatchSource:0}: Error finding container bccac22a4af094e7f341de9271910388d0bafe43225a904229ad1bc9cf2322bc: Status 404 returned error can't find the container with id bccac22a4af094e7f341de9271910388d0bafe43225a904229ad1bc9cf2322bc
Oct 30 13:32:58.535315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:58.535290    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"]
Oct 30 13:32:58.630476 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:58.630440    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4" event={"ID":"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34","Type":"ContainerStarted","Data":"bccac22a4af094e7f341de9271910388d0bafe43225a904229ad1bc9cf2322bc"}
Oct 30 13:32:58.908739 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:58.908715    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:32:58.908853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:58.908758    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj" podUID="30fc4668-2aed-4383-b5b1-092a5f3411a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 13:32:59.677299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:59.673368    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4" event={"ID":"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34","Type":"ContainerStarted","Data":"ae05b3fe4231bd0f19ab795bb41facfb7304ec99e7f60526d1e871031a8e0d1a"}
Oct 30 13:32:59.677299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:59.673707    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:32:59.715735 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:59.715685    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4" podStartSLOduration=2.341500731 podStartE2EDuration="2.715671324s" totalImagesPullingTime="374.170593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:32:59.712311714 +0000 UTC m=+2821.295741407" watchObservedRunningTime="2026-10-30 13:32:59.715671324 +0000 UTC m=+2821.299101016"
Oct 30 13:32:59.889587 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:59.889263    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"]
Oct 30 13:32:59.897714 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:59.895612    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:32:59.902198 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:59.899594    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"]
Oct 30 13:32:59.906888 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:59.906861    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:32:59.907760 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:59.907061    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj" podUID="30fc4668-2aed-4383-b5b1-092a5f3411a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 13:32:59.917404 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:59.917383    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-03b5b3\"/\"default-dockercfg-l5ktf\""
Oct 30 13:32:59.917570 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:59.917551    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-03b5b3\"/\"auth-test-model-03b5b3-kserve-self-signed-certs\""
Oct 30 13:32:59.917944 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:59.917918    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-03b5b3\"/\"openshift-service-ca.crt\""
Oct 30 13:32:59.918122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:32:59.918111    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-03b5b3\"/\"kube-root-ca.crt\""
Oct 30 13:33:00.017348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.017317    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-model-cache\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.017534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.017354    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-home\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.017534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.017440    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-tmp-dir\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.017534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.017457    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-tls-certs\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.017534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.017494    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-dshm\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.017673 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.017614    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vb9t\" (UniqueName: \"kubernetes.io/projected/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-kube-api-access-5vb9t\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.120428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.119674    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-model-cache\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.120428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.119711    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-home\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.120428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.119801    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-tmp-dir\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.120428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.119821    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-tls-certs\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.120428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.119858    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-dshm\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.120428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.119975    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vb9t\" (UniqueName: \"kubernetes.io/projected/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-kube-api-access-5vb9t\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.120815 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.120560    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-model-cache\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.120815 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.120725    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-home\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.120896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.120874    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-tmp-dir\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.123365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.123339    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-dshm\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.123626 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.123607    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-tls-certs\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.160324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.159061    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vb9t\" (UniqueName: \"kubernetes.io/projected/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-kube-api-access-5vb9t\") pod \"auth-test-model-03b5b3-kserve-7d76c5c749-b5krd\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") " pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.216301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.214700    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:00.690323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.689516    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"]
Oct 30 13:33:00.692169 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:33:00.692145    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42dbf1d0_1dc0_484e_a3e6_0e9ba1c912a1.slice/crio-8fd5fb509ab3fc1ae50588176bd70dc71a0f1cbe4461cf70d6f984308448f67e WatchSource:0}: Error finding container 8fd5fb509ab3fc1ae50588176bd70dc71a0f1cbe4461cf70d6f984308448f67e: Status 404 returned error can't find the container with id 8fd5fb509ab3fc1ae50588176bd70dc71a0f1cbe4461cf70d6f984308448f67e
Oct 30 13:33:00.906868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.906832    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:33:00.907053 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:00.906895    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj" podUID="30fc4668-2aed-4383-b5b1-092a5f3411a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 13:33:01.709806 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:01.709770    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd" event={"ID":"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1","Type":"ContainerStarted","Data":"c752029b0798b839e8ce02f71a0b1d6cbbf24c84ab29937523fa8cec95a3f812"}
Oct 30 13:33:01.710355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:01.709814    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd" event={"ID":"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1","Type":"ContainerStarted","Data":"8fd5fb509ab3fc1ae50588176bd70dc71a0f1cbe4461cf70d6f984308448f67e"}
Oct 30 13:33:01.778296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:01.778215    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd" podStartSLOduration=2.778192024 podStartE2EDuration="2.778192024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:01.755952052 +0000 UTC m=+2823.339381747" watchObservedRunningTime="2026-10-30 13:33:01.778192024 +0000 UTC m=+2823.361621720"
Oct 30 13:33:01.906782 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:01.906747    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:33:01.907076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:01.907047    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj" podUID="30fc4668-2aed-4383-b5b1-092a5f3411a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 13:33:02.215353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:02.215327    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:02.906821 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:02.906790    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:33:02.907213 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:02.906842    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj" podUID="30fc4668-2aed-4383-b5b1-092a5f3411a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": dial tcp 10.134.0.189:15021: connect: connection refused"
Oct 30 13:33:03.998220 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:03.998170    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:33:03.998729 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:03.998613    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:33:03.999572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:03.999549    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:33:05.693621 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:05.693594    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:33:09.886702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:09.886669    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"]
Oct 30 13:33:10.215926 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:10.215895    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:10.229663 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:10.229636    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:10.804264 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:10.804232    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:33:18.649375 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:33:18.649333    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:33:21.836552 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:21.836517    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"]
Oct 30 13:33:22.066349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.065210    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"]
Oct 30 13:33:22.066349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.065326    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"]
Oct 30 13:33:22.066976 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.066950    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.193834 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.193723    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.194111 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.194082    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74101d0-b434-47ce-99e5-351977da59ff-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.194258 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.194242    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkzrn\" (UniqueName: \"kubernetes.io/projected/b74101d0-b434-47ce-99e5-351977da59ff-kube-api-access-jkzrn\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.194523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.194498    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.194704 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.194682    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74101d0-b434-47ce-99e5-351977da59ff-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.194872 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.194850    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.195157 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.195123    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.195360 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.195330    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.195568 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.195546    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74101d0-b434-47ce-99e5-351977da59ff-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.210148 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.210121    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"]
Oct 30 13:33:22.210324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.210211    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"
Oct 30 13:33:22.217002 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.216977    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"]
Oct 30 13:33:22.222574 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.222551    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"
Oct 30 13:33:22.224007 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.223986    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"]
Oct 30 13:33:22.226736 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.226711    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3c1b28\""
Oct 30 13:33:22.226982 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.226957    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3c1b28-dockercfg-7j75k\""
Oct 30 13:33:22.297037 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297007    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.297244 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297068    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74101d0-b434-47ce-99e5-351977da59ff-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.297244 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297093    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tckc8\" (UniqueName: \"kubernetes.io/projected/01f1034a-cc31-43bc-91d4-8c0fc67e5960-kube-api-access-tckc8\") pod \"payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl\" (UID: \"01f1034a-cc31-43bc-91d4-8c0fc67e5960\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"
Oct 30 13:33:22.297244 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297131    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.297244 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297147    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f1034a-cc31-43bc-91d4-8c0fc67e5960-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl\" (UID: \"01f1034a-cc31-43bc-91d4-8c0fc67e5960\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"
Oct 30 13:33:22.297448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297391    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.297448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297411    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-istio-data\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.297448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297431    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.297448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297445    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-workload-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.297728 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297708    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-workload-certs\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.297767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297730    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74101d0-b434-47ce-99e5-351977da59ff-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.297804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297774    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n\" (UID: \"d3f79cd8-b784-4ef6-9a74-69a815c6e8a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"
Oct 30 13:33:22.297839 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297805    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dczn5\" (UniqueName: \"kubernetes.io/projected/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8-kube-api-access-dczn5\") pod \"payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n\" (UID: \"d3f79cd8-b784-4ef6-9a74-69a815c6e8a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"
Oct 30 13:33:22.297877 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297841    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74101d0-b434-47ce-99e5-351977da59ff-istiod-ca-cert\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.297977 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297956    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.298312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.297998    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74101d0-b434-47ce-99e5-351977da59ff-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.298312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.298020    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkzrn\" (UniqueName: \"kubernetes.io/projected/b74101d0-b434-47ce-99e5-351977da59ff-kube-api-access-jkzrn\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.298455 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.298340    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-credential-socket\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.299944 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.299878    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-istio-envoy\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.300473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.300446    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74101d0-b434-47ce-99e5-351977da59ff-istio-podinfo\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.319611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.319583    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkzrn\" (UniqueName: \"kubernetes.io/projected/b74101d0-b434-47ce-99e5-351977da59ff-kube-api-access-jkzrn\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.320087 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.320064    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74101d0-b434-47ce-99e5-351977da59ff-istio-token\") pod \"e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") " pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.399425 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.399390    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n\" (UID: \"d3f79cd8-b784-4ef6-9a74-69a815c6e8a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"
Oct 30 13:33:22.399693 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.399435    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dczn5\" (UniqueName: \"kubernetes.io/projected/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8-kube-api-access-dczn5\") pod \"payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n\" (UID: \"d3f79cd8-b784-4ef6-9a74-69a815c6e8a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"
Oct 30 13:33:22.399693 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.399599    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tckc8\" (UniqueName: \"kubernetes.io/projected/01f1034a-cc31-43bc-91d4-8c0fc67e5960-kube-api-access-tckc8\") pod \"payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl\" (UID: \"01f1034a-cc31-43bc-91d4-8c0fc67e5960\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"
Oct 30 13:33:22.399693 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.399660    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f1034a-cc31-43bc-91d4-8c0fc67e5960-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl\" (UID: \"01f1034a-cc31-43bc-91d4-8c0fc67e5960\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"
Oct 30 13:33:22.400309 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.400244    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n\" (UID: \"d3f79cd8-b784-4ef6-9a74-69a815c6e8a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"
Oct 30 13:33:22.400408 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.400365    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f1034a-cc31-43bc-91d4-8c0fc67e5960-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl\" (UID: \"01f1034a-cc31-43bc-91d4-8c0fc67e5960\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"
Oct 30 13:33:22.420820 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.420782    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tckc8\" (UniqueName: \"kubernetes.io/projected/01f1034a-cc31-43bc-91d4-8c0fc67e5960-kube-api-access-tckc8\") pod \"payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl\" (UID: \"01f1034a-cc31-43bc-91d4-8c0fc67e5960\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"
Oct 30 13:33:22.420953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.420884    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dczn5\" (UniqueName: \"kubernetes.io/projected/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8-kube-api-access-dczn5\") pod \"payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n\" (UID: \"d3f79cd8-b784-4ef6-9a74-69a815c6e8a8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"
Oct 30 13:33:22.546004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.545971    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:22.548126 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.548098    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"
Oct 30 13:33:22.549663 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.549588    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"
Oct 30 13:33:22.915190 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.915168    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"]
Oct 30 13:33:22.918182 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:33:22.918156    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f79cd8_b784_4ef6_9a74_69a815c6e8a8.slice/crio-ae1eae184c862dd85721c2c0f779a7a60bf7d76393cc480d5ba7a5fc322fc262 WatchSource:0}: Error finding container ae1eae184c862dd85721c2c0f779a7a60bf7d76393cc480d5ba7a5fc322fc262: Status 404 returned error can't find the container with id ae1eae184c862dd85721c2c0f779a7a60bf7d76393cc480d5ba7a5fc322fc262
Oct 30 13:33:22.918721 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:33:22.918699    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f1034a_cc31_43bc_91d4_8c0fc67e5960.slice/crio-89a55d3fa262917bdc6fc0c69339827f304acfd1fb481f1cc5cca53737929c6f WatchSource:0}: Error finding container 89a55d3fa262917bdc6fc0c69339827f304acfd1fb481f1cc5cca53737929c6f: Status 404 returned error can't find the container with id 89a55d3fa262917bdc6fc0c69339827f304acfd1fb481f1cc5cca53737929c6f
Oct 30 13:33:22.921006 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.920985    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"]
Oct 30 13:33:22.925808 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.925778    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"]
Oct 30 13:33:22.933046 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:33:22.933016    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74101d0_b434_47ce_99e5_351977da59ff.slice/crio-00fb235ccd3b9e5f538386ed7d49b3b0b1c77f4bb34979da9321312e5b0c058e WatchSource:0}: Error finding container 00fb235ccd3b9e5f538386ed7d49b3b0b1c77f4bb34979da9321312e5b0c058e: Status 404 returned error can't find the container with id 00fb235ccd3b9e5f538386ed7d49b3b0b1c77f4bb34979da9321312e5b0c058e
Oct 30 13:33:22.940743 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.940700    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:33:22.940859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.940783    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:33:22.940859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:22.940835    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:33:23.211546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:23.211362    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl" event={"ID":"01f1034a-cc31-43bc-91d4-8c0fc67e5960","Type":"ContainerStarted","Data":"d16a7c588fa50f4f52ed027cda22c53cd17b743257f57419b7223e2853897648"}
Oct 30 13:33:23.211546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:23.211420    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"
Oct 30 13:33:23.211546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:23.211445    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl" event={"ID":"01f1034a-cc31-43bc-91d4-8c0fc67e5960","Type":"ContainerStarted","Data":"89a55d3fa262917bdc6fc0c69339827f304acfd1fb481f1cc5cca53737929c6f"}
Oct 30 13:33:23.214120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:23.214088    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8" event={"ID":"b74101d0-b434-47ce-99e5-351977da59ff","Type":"ContainerStarted","Data":"7fbc76547757f894b1adad2d07c43134239780e9b76e68cab098a39f19c8171e"}
Oct 30 13:33:23.214285 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:23.214149    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8" event={"ID":"b74101d0-b434-47ce-99e5-351977da59ff","Type":"ContainerStarted","Data":"00fb235ccd3b9e5f538386ed7d49b3b0b1c77f4bb34979da9321312e5b0c058e"}
Oct 30 13:33:23.216832 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:23.216801    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n" event={"ID":"d3f79cd8-b784-4ef6-9a74-69a815c6e8a8","Type":"ContainerStarted","Data":"707286ec786736b8b5f55628f6379022f1ad10a0f654c8dd6c6462cbad4addce"}
Oct 30 13:33:23.216959 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:23.216853    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n" event={"ID":"d3f79cd8-b784-4ef6-9a74-69a815c6e8a8","Type":"ContainerStarted","Data":"ae1eae184c862dd85721c2c0f779a7a60bf7d76393cc480d5ba7a5fc322fc262"}
Oct 30 13:33:23.216959 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:23.216887    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"
Oct 30 13:33:23.257297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:23.257216    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl" podStartSLOduration=2.257200252 podStartE2EDuration="2.257200252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:23.252649466 +0000 UTC m=+2844.836079160" watchObservedRunningTime="2026-10-30 13:33:23.257200252 +0000 UTC m=+2844.840629956"
Oct 30 13:33:23.302308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:23.302144    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8" podStartSLOduration=2.302123516 podStartE2EDuration="2.302123516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:23.299768843 +0000 UTC m=+2844.883198535" watchObservedRunningTime="2026-10-30 13:33:23.302123516 +0000 UTC m=+2844.885553210"
Oct 30 13:33:23.337343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:23.337289    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n" podStartSLOduration=1.33725831 podStartE2EDuration="1.33725831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:23.335922844 +0000 UTC m=+2844.919352562" watchObservedRunningTime="2026-10-30 13:33:23.33725831 +0000 UTC m=+2844.920688002"
Oct 30 13:33:23.546823 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:23.546795    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:24.561295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:24.560348    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 13:33:24.561295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:24.560410    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8" podUID="b74101d0-b434-47ce-99e5-351977da59ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 13:33:25.282605 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.282565    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj" podUID="7d032dc0-e641-447b-bce2-c46b5968cb6f" containerName="main" containerID="cri-o://8994f55d7adab87dbf363d2e45298b08091fa4db8bea56395ec1574a02482d71" gracePeriod=45
Oct 30 13:33:25.572348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.571751    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 13:33:25.572348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.571843    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8" podUID="b74101d0-b434-47ce-99e5-351977da59ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 13:33:25.727829 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.727804    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:33:25.901263 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.901175    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d032dc0-e641-447b-bce2-c46b5968cb6f-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d032dc0-e641-447b-bce2-c46b5968cb6f-tls-certs\") pod \"7d032dc0-e641-447b-bce2-c46b5968cb6f\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") "
Oct 30 13:33:25.901263 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.901232    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-dshm\") pod \"7d032dc0-e641-447b-bce2-c46b5968cb6f\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") "
Oct 30 13:33:25.901263 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.901259    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-model-cache\") pod \"7d032dc0-e641-447b-bce2-c46b5968cb6f\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") "
Oct 30 13:33:25.901591 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.901352    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-tmp-dir\") pod \"7d032dc0-e641-447b-bce2-c46b5968cb6f\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") "
Oct 30 13:33:25.901591 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.901496    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d032dc0-e641-447b-bce2-c46b5968cb6f-kube-api-access-ggw6c\" (UniqueName: \"kubernetes.io/projected/7d032dc0-e641-447b-bce2-c46b5968cb6f-kube-api-access-ggw6c\") pod \"7d032dc0-e641-447b-bce2-c46b5968cb6f\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") "
Oct 30 13:33:25.901591 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.901526    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-home\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-home\") pod \"7d032dc0-e641-447b-bce2-c46b5968cb6f\" (UID: \"7d032dc0-e641-447b-bce2-c46b5968cb6f\") "
Oct 30 13:33:25.901758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.901606    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-tmp-dir" pod "7d032dc0-e641-447b-bce2-c46b5968cb6f" (UID: "7d032dc0-e641-447b-bce2-c46b5968cb6f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:33:25.901871 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.901849    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-model-cache" pod "7d032dc0-e641-447b-bce2-c46b5968cb6f" (UID: "7d032dc0-e641-447b-bce2-c46b5968cb6f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:33:25.901924 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.901866    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-home" pod "7d032dc0-e641-447b-bce2-c46b5968cb6f" (UID: "7d032dc0-e641-447b-bce2-c46b5968cb6f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:33:25.902258 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.902241    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:33:25.902344 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.902260    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:33:25.902344 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.902294    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:33:25.906071 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.905937    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d032dc0-e641-447b-bce2-c46b5968cb6f-tls-certs" pod "7d032dc0-e641-447b-bce2-c46b5968cb6f" (UID: "7d032dc0-e641-447b-bce2-c46b5968cb6f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:33:25.906211 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.906101    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-dshm" pod "7d032dc0-e641-447b-bce2-c46b5968cb6f" (UID: "7d032dc0-e641-447b-bce2-c46b5968cb6f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:33:25.906211 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:25.906145    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d032dc0-e641-447b-bce2-c46b5968cb6f-kube-api-access-ggw6c" pod "7d032dc0-e641-447b-bce2-c46b5968cb6f" (UID: "7d032dc0-e641-447b-bce2-c46b5968cb6f"). InnerVolumeSpecName "kube-api-access-ggw6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:33:26.004168 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.004141    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggw6c\" (UniqueName: \"kubernetes.io/projected/7d032dc0-e641-447b-bce2-c46b5968cb6f-kube-api-access-ggw6c\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:33:26.004405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.004391    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d032dc0-e641-447b-bce2-c46b5968cb6f-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:33:26.004469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.004462    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d032dc0-e641-447b-bce2-c46b5968cb6f-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:33:26.277298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.277127    5893 generic.go:356] "Generic (PLEG): container finished" podID="7d032dc0-e641-447b-bce2-c46b5968cb6f" containerID="8994f55d7adab87dbf363d2e45298b08091fa4db8bea56395ec1574a02482d71" exitCode=0
Oct 30 13:33:26.277490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.277301    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj" event={"ID":"7d032dc0-e641-447b-bce2-c46b5968cb6f","Type":"ContainerDied","Data":"8994f55d7adab87dbf363d2e45298b08091fa4db8bea56395ec1574a02482d71"}
Oct 30 13:33:26.277490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.277342    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj" event={"ID":"7d032dc0-e641-447b-bce2-c46b5968cb6f","Type":"ContainerDied","Data":"025395ca47af642c0e3fa7c7a17a9bb381be21395b75dced87001a25db4d932e"}
Oct 30 13:33:26.277490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.277363    5893 scope.go:122] "RemoveContainer" containerID="8994f55d7adab87dbf363d2e45298b08091fa4db8bea56395ec1574a02482d71"
Oct 30 13:33:26.277490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.277460    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"
Oct 30 13:33:26.291145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.291120    5893 scope.go:122] "RemoveContainer" containerID="8994f55d7adab87dbf363d2e45298b08091fa4db8bea56395ec1574a02482d71"
Oct 30 13:33:26.291517 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:33:26.291491    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8994f55d7adab87dbf363d2e45298b08091fa4db8bea56395ec1574a02482d71\": container with ID starting with 8994f55d7adab87dbf363d2e45298b08091fa4db8bea56395ec1574a02482d71 not found: ID does not exist" containerID="8994f55d7adab87dbf363d2e45298b08091fa4db8bea56395ec1574a02482d71"
Oct 30 13:33:26.291585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.291531    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8994f55d7adab87dbf363d2e45298b08091fa4db8bea56395ec1574a02482d71"} err="failed to get container status \"8994f55d7adab87dbf363d2e45298b08091fa4db8bea56395ec1574a02482d71\": rpc error: code = NotFound desc = could not find container \"8994f55d7adab87dbf363d2e45298b08091fa4db8bea56395ec1574a02482d71\": container with ID starting with 8994f55d7adab87dbf363d2e45298b08091fa4db8bea56395ec1574a02482d71 not found: ID does not exist"
Oct 30 13:33:26.325456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.325422    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"]
Oct 30 13:33:26.335529 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.335492    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-96ab73/auth-test-model-96ab73-kserve-5b689558d5-2rdbj"]
Oct 30 13:33:26.533751 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.533674    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d032dc0-e641-447b-bce2-c46b5968cb6f" path="/var/lib/kubelet/pods/7d032dc0-e641-447b-bce2-c46b5968cb6f/volumes"
Oct 30 13:33:26.562332 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.562299    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 13:33:26.562622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:26.562593    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8" podUID="b74101d0-b434-47ce-99e5-351977da59ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 13:33:27.560298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:27.559639    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 13:33:27.560298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:27.559709    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8" podUID="b74101d0-b434-47ce-99e5-351977da59ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 13:33:28.552394 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:28.552361    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 13:33:28.552668 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:28.552644    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8" podUID="b74101d0-b434-47ce-99e5-351977da59ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 13:33:29.565300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:29.561821    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Oct 30 13:33:29.565300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:29.561891    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8" podUID="b74101d0-b434-47ce-99e5-351977da59ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Oct 30 13:33:30.552484 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:30.552452    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:30.552993 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:30.552972    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:30.553824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:30.553802    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:33:34.229250 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:34.229218    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"
Oct 30 13:33:34.231119 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:34.231094    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"
Oct 30 13:33:47.920081 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:47.920032    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"]
Oct 30 13:33:47.920957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:47.920933    5893 cpu_manager.go:414] "containerMap: removing container" podUID="7d032dc0-e641-447b-bce2-c46b5968cb6f" containerName="main"
Oct 30 13:33:47.920957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:47.920950    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d032dc0-e641-447b-bce2-c46b5968cb6f" containerName="main"
Oct 30 13:33:47.921090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:47.921080    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d032dc0-e641-447b-bce2-c46b5968cb6f" containerName="main"
Oct 30 13:33:47.924951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:47.924929    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"
Oct 30 13:33:47.980515 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:47.980483    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-78543b-dockercfg-q5khc\""
Oct 30 13:33:47.982541 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:47.982511    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-78543b\""
Oct 30 13:33:47.989936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:47.989906    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"]
Oct 30 13:33:48.078295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:48.075116    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467105e6-571f-478b-8324-9e61bc980757-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8\" (UID: \"467105e6-571f-478b-8324-9e61bc980757\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"
Oct 30 13:33:48.078295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:48.075238    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8nxm\" (UniqueName: \"kubernetes.io/projected/467105e6-571f-478b-8324-9e61bc980757-kube-api-access-k8nxm\") pod \"payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8\" (UID: \"467105e6-571f-478b-8324-9e61bc980757\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"
Oct 30 13:33:48.179773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:48.178291    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467105e6-571f-478b-8324-9e61bc980757-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8\" (UID: \"467105e6-571f-478b-8324-9e61bc980757\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"
Oct 30 13:33:48.179773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:48.178389    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8nxm\" (UniqueName: \"kubernetes.io/projected/467105e6-571f-478b-8324-9e61bc980757-kube-api-access-k8nxm\") pod \"payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8\" (UID: \"467105e6-571f-478b-8324-9e61bc980757\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"
Oct 30 13:33:48.179773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:48.179673    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467105e6-571f-478b-8324-9e61bc980757-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8\" (UID: \"467105e6-571f-478b-8324-9e61bc980757\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"
Oct 30 13:33:48.218064 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:48.218033    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8nxm\" (UniqueName: \"kubernetes.io/projected/467105e6-571f-478b-8324-9e61bc980757-kube-api-access-k8nxm\") pod \"payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8\" (UID: \"467105e6-571f-478b-8324-9e61bc980757\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"
Oct 30 13:33:48.285456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:48.285424    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"
Oct 30 13:33:48.651069 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:33:48.651003    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:33:48.661623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:48.661597    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"]
Oct 30 13:33:48.664770 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:33:48.664739    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod467105e6_571f_478b_8324_9e61bc980757.slice/crio-3ff0b0c38c78df646d0fca2d00bfccd9b5a64fab10038d732764fcec6f8c936b WatchSource:0}: Error finding container 3ff0b0c38c78df646d0fca2d00bfccd9b5a64fab10038d732764fcec6f8c936b: Status 404 returned error can't find the container with id 3ff0b0c38c78df646d0fca2d00bfccd9b5a64fab10038d732764fcec6f8c936b
Oct 30 13:33:48.723214 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:48.723181    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8" event={"ID":"467105e6-571f-478b-8324-9e61bc980757","Type":"ContainerStarted","Data":"3ff0b0c38c78df646d0fca2d00bfccd9b5a64fab10038d732764fcec6f8c936b"}
Oct 30 13:33:49.735950 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:49.735904    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8" event={"ID":"467105e6-571f-478b-8324-9e61bc980757","Type":"ContainerStarted","Data":"354d364e7c5c4c7c9a8d09992df4c5cc7d51169d6b4900007294922269b3e38f"}
Oct 30 13:33:49.736584 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:49.736555    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"
Oct 30 13:33:49.777780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:49.777730    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8" podStartSLOduration=2.777713205 podStartE2EDuration="2.777713205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:49.776338726 +0000 UTC m=+2871.359768424" watchObservedRunningTime="2026-10-30 13:33:49.777713205 +0000 UTC m=+2871.361142936"
Oct 30 13:33:52.167783 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.167722    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"]
Oct 30 13:33:52.178379 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.178341    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.208819 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.208789    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3b47e7-openshift-default-dockercfg-fkvcj\""
Oct 30 13:33:52.226329 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.226292    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.226759 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.226736    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnt6m\" (UniqueName: \"kubernetes.io/projected/45a767a0-a88a-4a84-b0e1-c918badbd3e9-kube-api-access-mnt6m\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.257423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.226993    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.257423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.227097    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.257423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.227147    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.257423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.251057    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"]
Oct 30 13:33:52.257423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.251235    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.257423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.251306    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.257423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.251366    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.257423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.251446    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.353912 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.353859    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.354245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.354221    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.354562 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.354539    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.354727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.354710    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnt6m\" (UniqueName: \"kubernetes.io/projected/45a767a0-a88a-4a84-b0e1-c918badbd3e9-kube-api-access-mnt6m\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.354956 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.354938    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.355167 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.355151    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.355338 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.355320    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.355519 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.355503    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.355655 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.355638    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.356789 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.356766    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.357249 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.357212    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.361425 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.359653    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.361583 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.359665    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.361583 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.359994    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.367546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.367521    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.376127 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.367753    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.402297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.392870    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnt6m\" (UniqueName: \"kubernetes.io/projected/45a767a0-a88a-4a84-b0e1-c918badbd3e9-kube-api-access-mnt6m\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.416164 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.415892    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:52.635349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:52.634045    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:53.029956 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:53.029922    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"]
Oct 30 13:33:53.145855 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:33:53.119474    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a767a0_a88a_4a84_b0e1_c918badbd3e9.slice/crio-5a340da65b3b0fbff5ec7221e51e1809241a8d4bf88811fc7d18419ac058c0ae WatchSource:0}: Error finding container 5a340da65b3b0fbff5ec7221e51e1809241a8d4bf88811fc7d18419ac058c0ae: Status 404 returned error can't find the container with id 5a340da65b3b0fbff5ec7221e51e1809241a8d4bf88811fc7d18419ac058c0ae
Oct 30 13:33:53.145855 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:53.123179    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:33:53.145855 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:53.123250    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:33:53.145855 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:53.123301    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:33:53.784295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:53.784093    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w" event={"ID":"45a767a0-a88a-4a84-b0e1-c918badbd3e9","Type":"ContainerStarted","Data":"1484fa554faadcf420790790e30c2bad1efaf2cd2a5d01c1b45044a8cb42731f"}
Oct 30 13:33:53.784295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:53.784165    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w" event={"ID":"45a767a0-a88a-4a84-b0e1-c918badbd3e9","Type":"ContainerStarted","Data":"5a340da65b3b0fbff5ec7221e51e1809241a8d4bf88811fc7d18419ac058c0ae"}
Oct 30 13:33:53.950964 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:53.950852    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w" podStartSLOduration=1.950830517 podStartE2EDuration="1.950830517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:53.943791542 +0000 UTC m=+2875.527221247" watchObservedRunningTime="2026-10-30 13:33:53.950830517 +0000 UTC m=+2875.534260211"
Oct 30 13:33:54.634568 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:54.634530    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:54.642033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:54.642005    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:54.797171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:54.796660    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:54.847954 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:54.797709    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:33:58.929585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:58.929551    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"]
Oct 30 13:33:59.079556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.075942    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"]
Oct 30 13:33:59.079556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.076008    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"]
Oct 30 13:33:59.104205 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.104175    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"
Oct 30 13:33:59.215654 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.146305    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3b47e7-dockercfg-f7p4x\""
Oct 30 13:33:59.215654 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.146392    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3b47e7\""
Oct 30 13:33:59.215654 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.190168    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rblx5\" (UniqueName: \"kubernetes.io/projected/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d-kube-api-access-rblx5\") pod \"payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj\" (UID: \"8ae38ffb-3bee-4e1d-ad7a-5be201a7771d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"
Oct 30 13:33:59.215654 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.190220    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj\" (UID: \"8ae38ffb-3bee-4e1d-ad7a-5be201a7771d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"
Oct 30 13:33:59.384543 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.333621    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rblx5\" (UniqueName: \"kubernetes.io/projected/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d-kube-api-access-rblx5\") pod \"payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj\" (UID: \"8ae38ffb-3bee-4e1d-ad7a-5be201a7771d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"
Oct 30 13:33:59.384543 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.333665    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj\" (UID: \"8ae38ffb-3bee-4e1d-ad7a-5be201a7771d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"
Oct 30 13:33:59.384543 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.334836    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj\" (UID: \"8ae38ffb-3bee-4e1d-ad7a-5be201a7771d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"
Oct 30 13:33:59.409471 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.402666    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"]
Oct 30 13:33:59.409471 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.402775    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"
Oct 30 13:33:59.531180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.531150    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rblx5\" (UniqueName: \"kubernetes.io/projected/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d-kube-api-access-rblx5\") pod \"payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj\" (UID: \"8ae38ffb-3bee-4e1d-ad7a-5be201a7771d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"
Oct 30 13:33:59.547957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.537740    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlhld\" (UniqueName: \"kubernetes.io/projected/914da95d-9ca6-4756-b909-9074526ff434-kube-api-access-dlhld\") pod \"payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv\" (UID: \"914da95d-9ca6-4756-b909-9074526ff434\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"
Oct 30 13:33:59.547957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.537867    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914da95d-9ca6-4756-b909-9074526ff434-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv\" (UID: \"914da95d-9ca6-4756-b909-9074526ff434\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"
Oct 30 13:33:59.639652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.639618    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlhld\" (UniqueName: \"kubernetes.io/projected/914da95d-9ca6-4756-b909-9074526ff434-kube-api-access-dlhld\") pod \"payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv\" (UID: \"914da95d-9ca6-4756-b909-9074526ff434\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"
Oct 30 13:33:59.643961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.639714    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914da95d-9ca6-4756-b909-9074526ff434-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv\" (UID: \"914da95d-9ca6-4756-b909-9074526ff434\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"
Oct 30 13:33:59.643961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.640639    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914da95d-9ca6-4756-b909-9074526ff434-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv\" (UID: \"914da95d-9ca6-4756-b909-9074526ff434\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"
Oct 30 13:33:59.662782 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.662751    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlhld\" (UniqueName: \"kubernetes.io/projected/914da95d-9ca6-4756-b909-9074526ff434-kube-api-access-dlhld\") pod \"payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv\" (UID: \"914da95d-9ca6-4756-b909-9074526ff434\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"
Oct 30 13:33:59.747880 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.747788    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"
Oct 30 13:33:59.851347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:33:59.850971    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"
Oct 30 13:34:00.955187 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:00.953078    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"
Oct 30 13:34:01.473486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:01.473437    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"]
Oct 30 13:34:01.490239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:01.490154    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"]
Oct 30 13:34:02.075645 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:02.075546    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj" event={"ID":"8ae38ffb-3bee-4e1d-ad7a-5be201a7771d","Type":"ContainerStarted","Data":"af3df3f8dec993c875a7288571dda4e756e058d94e4a20211e11adcd05f8a66c"}
Oct 30 13:34:02.075645 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:02.075612    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj" event={"ID":"8ae38ffb-3bee-4e1d-ad7a-5be201a7771d","Type":"ContainerStarted","Data":"d1c471b2aafc48b224a625d852d562b09267b31c8738da1660d151ee9fddadec"}
Oct 30 13:34:02.076261 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:02.075674    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"
Oct 30 13:34:02.077842 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:02.077813    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv" event={"ID":"914da95d-9ca6-4756-b909-9074526ff434","Type":"ContainerStarted","Data":"2bb7c0d6356441ac57b3f69cc34b253ba1c2500503274ac607ef1be19cfb6fb2"}
Oct 30 13:34:02.087164 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:02.077865    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv" event={"ID":"914da95d-9ca6-4756-b909-9074526ff434","Type":"ContainerStarted","Data":"10cce193c1f11dad2ca7af56f2527405ea06884b12ad479108f1f2c73cf0c56a"}
Oct 30 13:34:02.087164 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:02.077908    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"
Oct 30 13:34:02.119512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:02.119461    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj" podStartSLOduration=4.11944804 podStartE2EDuration="4.11944804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:02.11744591 +0000 UTC m=+2883.700875603" watchObservedRunningTime="2026-10-30 13:34:02.11944804 +0000 UTC m=+2883.702877737"
Oct 30 13:34:02.157924 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:02.157872    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv" podStartSLOduration=4.15785855 podStartE2EDuration="4.15785855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:02.153411841 +0000 UTC m=+2883.736841553" watchObservedRunningTime="2026-10-30 13:34:02.15785855 +0000 UTC m=+2883.741288282"
Oct 30 13:34:03.492744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:03.488169    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": read tcp 10.134.0.2:56174->10.134.0.184:15021: read: connection reset by peer" start-of-body=
Oct 30 13:34:03.492744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:03.488236    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": read tcp 10.134.0.2:56174->10.134.0.184:15021: read: connection reset by peer"
Oct 30 13:34:04.098698 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:04.098667    5893 generic.go:356] "Generic (PLEG): container finished" podID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerID="6760e5c07c8e77766167a484dccf47838318d578fd146c9395330692f1de6431" exitCode=0
Oct 30 13:34:04.098878 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:04.098743    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" event={"ID":"8b1530d7-d75e-4d05-9fda-46f3a07b60c1","Type":"ContainerDied","Data":"6760e5c07c8e77766167a484dccf47838318d578fd146c9395330692f1de6431"}
Oct 30 13:34:04.099295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:04.099253    5893 scope.go:122] "RemoveContainer" containerID="6760e5c07c8e77766167a484dccf47838318d578fd146c9395330692f1de6431"
Oct 30 13:34:04.101441 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:04.101412    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:34:04.102422 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:04.102387    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:34:04.102554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:04.102441    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:34:05.119750 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:05.119172    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" event={"ID":"8b1530d7-d75e-4d05-9fda-46f3a07b60c1","Type":"ContainerStarted","Data":"43ec74448547cda39d22fbe5e3d1b0b1bc4e335bf70251a9bcb78a7f7170cf96"}
Oct 30 13:34:05.395513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:05.395423    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:34:05.395685 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:05.395632    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 13:34:05.395741 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:05.395687    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 13:34:06.395597 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:06.395566    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 13:34:06.396103 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:06.395626    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 13:34:07.396134 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:07.395978    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 13:34:07.396134 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:07.396043    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 13:34:08.395681 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:08.395648    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 13:34:08.396413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:08.395861    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 13:34:09.413296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:09.407790    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 13:34:09.413296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:09.407878    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 13:34:10.395668 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:10.395626    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused" start-of-body=
Oct 30 13:34:10.395840 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:10.395700    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.184:15021/healthz/ready\": dial tcp 10.134.0.184:15021: connect: connection refused"
Oct 30 13:34:11.400010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:11.399981    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:34:11.400547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:11.400294    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:34:11.401087 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:11.401071    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:34:13.145477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:13.145445    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"
Oct 30 13:34:13.179510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:34:13.179474    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"
Oct 30 13:34:18.672124 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:34:18.672085    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:34:48.647585 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:34:48.647543    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:35:18.764952 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:35:18.764754    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:35:34.572581 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:34.572442    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"]
Oct 30 13:35:40.838539 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:40.838449    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:35:40.838539 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:40.838522    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w" podUID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:35:43.210118 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:43.210082    5893 generic.go:356] "Generic (PLEG): container finished" podID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerID="1484fa554faadcf420790790e30c2bad1efaf2cd2a5d01c1b45044a8cb42731f" exitCode=0
Oct 30 13:35:43.211217 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:43.210150    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w" event={"ID":"45a767a0-a88a-4a84-b0e1-c918badbd3e9","Type":"ContainerDied","Data":"1484fa554faadcf420790790e30c2bad1efaf2cd2a5d01c1b45044a8cb42731f"}
Oct 30 13:35:43.211217 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:43.210628    5893 scope.go:122] "RemoveContainer" containerID="1484fa554faadcf420790790e30c2bad1efaf2cd2a5d01c1b45044a8cb42731f"
Oct 30 13:35:43.212713 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:43.212685    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:35:43.212778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:43.212764    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:35:43.212841 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:43.212805    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:35:44.222026 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:44.221992    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w" event={"ID":"45a767a0-a88a-4a84-b0e1-c918badbd3e9","Type":"ContainerStarted","Data":"457d9b6276efd816731ecf5561cb5fbe208cd49b166648cfb1745fe8575429ff"}
Oct 30 13:35:44.634832 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:44.634743    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:35:44.635006 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:44.634984    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:35:44.635061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:44.635038    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w" podUID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 30 13:35:45.635315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:45.635240    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:35:45.636026 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:45.635358    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w" podUID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 30 13:35:46.635458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:46.635428    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:35:46.635952 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:46.635486    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w" podUID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 30 13:35:47.635712 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:47.635668    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:35:47.636402 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:47.636363    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w" podUID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 30 13:35:48.634875 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:48.634845    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:35:48.635061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:48.634909    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w" podUID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Oct 30 13:35:48.692801 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:35:48.692769    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:35:49.677908 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:49.677852    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:35:49.678739 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:49.678618    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:35:49.679704 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:49.679683    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:35:50.417513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.417458    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd" podUID="42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1" containerName="main" containerID="cri-o://c752029b0798b839e8ce02f71a0b1d6cbbf24c84ab29937523fa8cec95a3f812" gracePeriod=45
Oct 30 13:35:50.779455 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.779423    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:35:50.931371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.931338    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-kube-api-access-5vb9t\" (UniqueName: \"kubernetes.io/projected/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-kube-api-access-5vb9t\") pod \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") "
Oct 30 13:35:50.931554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.931431    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-dshm\") pod \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") "
Oct 30 13:35:50.931554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.931478    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-tls-certs\") pod \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") "
Oct 30 13:35:50.931554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.931545    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-tmp-dir\") pod \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") "
Oct 30 13:35:50.931730 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.931714    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-model-cache\") pod \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") "
Oct 30 13:35:50.931795 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.931745    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-home\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-home\") pod \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\" (UID: \"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1\") "
Oct 30 13:35:50.931932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.931896    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-tmp-dir" pod "42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1" (UID: "42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:50.932048 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.932023    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-model-cache" pod "42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1" (UID: "42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:50.932048 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.932036    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-home" pod "42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1" (UID: "42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:50.933019 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.932992    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:35:50.933019 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.933015    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:35:50.933091 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.933028    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:35:50.946412 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.946367    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-dshm" pod "42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1" (UID: "42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:50.946558 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.946470    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-kube-api-access-5vb9t" pod "42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1" (UID: "42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1"). InnerVolumeSpecName "kube-api-access-5vb9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:50.947637 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:50.947612    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-tls-certs" pod "42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1" (UID: "42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:51.034787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:51.034706    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vb9t\" (UniqueName: \"kubernetes.io/projected/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-kube-api-access-5vb9t\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:35:51.034787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:51.034734    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:35:51.034787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:51.034746    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:35:51.345053 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:51.344970    5893 generic.go:356] "Generic (PLEG): container finished" podID="42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1" containerID="c752029b0798b839e8ce02f71a0b1d6cbbf24c84ab29937523fa8cec95a3f812" exitCode=0
Oct 30 13:35:51.345315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:51.345260    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"
Oct 30 13:35:51.345693 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:51.345673    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd" event={"ID":"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1","Type":"ContainerDied","Data":"c752029b0798b839e8ce02f71a0b1d6cbbf24c84ab29937523fa8cec95a3f812"}
Oct 30 13:35:51.345793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:51.345708    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd" event={"ID":"42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1","Type":"ContainerDied","Data":"8fd5fb509ab3fc1ae50588176bd70dc71a0f1cbe4461cf70d6f984308448f67e"}
Oct 30 13:35:51.345793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:51.345726    5893 scope.go:122] "RemoveContainer" containerID="c752029b0798b839e8ce02f71a0b1d6cbbf24c84ab29937523fa8cec95a3f812"
Oct 30 13:35:51.367449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:51.364113    5893 scope.go:122] "RemoveContainer" containerID="c752029b0798b839e8ce02f71a0b1d6cbbf24c84ab29937523fa8cec95a3f812"
Oct 30 13:35:51.367449 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:35:51.366066    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c752029b0798b839e8ce02f71a0b1d6cbbf24c84ab29937523fa8cec95a3f812\": container with ID starting with c752029b0798b839e8ce02f71a0b1d6cbbf24c84ab29937523fa8cec95a3f812 not found: ID does not exist" containerID="c752029b0798b839e8ce02f71a0b1d6cbbf24c84ab29937523fa8cec95a3f812"
Oct 30 13:35:51.367449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:51.366106    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c752029b0798b839e8ce02f71a0b1d6cbbf24c84ab29937523fa8cec95a3f812"} err="failed to get container status \"c752029b0798b839e8ce02f71a0b1d6cbbf24c84ab29937523fa8cec95a3f812\": rpc error: code = NotFound desc = could not find container \"c752029b0798b839e8ce02f71a0b1d6cbbf24c84ab29937523fa8cec95a3f812\": container with ID starting with c752029b0798b839e8ce02f71a0b1d6cbbf24c84ab29937523fa8cec95a3f812 not found: ID does not exist"
Oct 30 13:35:51.396434 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:51.396407    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"]
Oct 30 13:35:51.406625 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:51.406594    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-03b5b3/auth-test-model-03b5b3-kserve-7d76c5c749-b5krd"]
Oct 30 13:35:52.535096 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:35:52.535065    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1" path="/var/lib/kubelet/pods/42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1/volumes"
Oct 30 13:36:01.321888 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:01.321854    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:36:01.335780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:01.335758    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:36:01.381839 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:01.381811    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:36:01.396687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:01.396666    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:36:18.648525 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:36:18.648461    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:36:30.809761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:30.809726    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"]
Oct 30 13:36:30.811209 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:30.809980    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" containerName="istio-proxy" containerID="cri-o://47ebcee4149f5122d1f56ec827bf0551e474652c23fa8a02a5d075f0a55c8dcd" gracePeriod=30
Oct 30 13:36:30.818317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:30.818286    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"]
Oct 30 13:36:30.818519 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:30.818498    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz" podUID="1e84f541-2caa-40dc-9b56-f320c72f5097" containerName="istio-proxy" containerID="cri-o://34e990fe417fe794072cafeb38ede2453d16da8d8f8b390fe9735a5adb817e00" gracePeriod=30
Oct 30 13:36:33.692848 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:33.692810    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:36:33.693360 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:33.692867    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz" podUID="1e84f541-2caa-40dc-9b56-f320c72f5097" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:36:35.259574 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:35.259535    5893 patch_prober.go:29] interesting pod/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:36:35.260070 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:35.259606    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:36:36.196879 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.196844    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:36:36.209799 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.209773    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:36:36.297541 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.297059    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"]
Oct 30 13:36:36.301369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.299432    5893 cpu_manager.go:414] "containerMap: removing container" podUID="1e84f541-2caa-40dc-9b56-f320c72f5097" containerName="istio-proxy"
Oct 30 13:36:36.301369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.299453    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e84f541-2caa-40dc-9b56-f320c72f5097" containerName="istio-proxy"
Oct 30 13:36:36.301369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.299491    5893 cpu_manager.go:414] "containerMap: removing container" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" containerName="istio-proxy"
Oct 30 13:36:36.301369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.299499    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" containerName="istio-proxy"
Oct 30 13:36:36.301369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.299516    5893 cpu_manager.go:414] "containerMap: removing container" podUID="42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1" containerName="main"
Oct 30 13:36:36.301369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.299525    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1" containerName="main"
Oct 30 13:36:36.301369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.299579    5893 cpu_manager.go:414] "containerMap: removing container" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" containerName="istio-proxy"
Oct 30 13:36:36.301369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.299586    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" containerName="istio-proxy"
Oct 30 13:36:36.301369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.299819    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e84f541-2caa-40dc-9b56-f320c72f5097" containerName="istio-proxy"
Oct 30 13:36:36.301369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.299837    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" containerName="istio-proxy"
Oct 30 13:36:36.301369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.299849    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="42dbf1d0-1dc0-484e-a3e6-0e9ba1c912a1" containerName="main"
Oct 30 13:36:36.307012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.306092    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.323560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.322310    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e84f541-2caa-40dc-9b56-f320c72f5097-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e84f541-2caa-40dc-9b56-f320c72f5097-istiod-ca-cert\") pod \"1e84f541-2caa-40dc-9b56-f320c72f5097\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") "
Oct 30 13:36:36.323560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.322487    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e84f541-2caa-40dc-9b56-f320c72f5097-kube-api-access-m5ngn\" (UniqueName: \"kubernetes.io/projected/1e84f541-2caa-40dc-9b56-f320c72f5097-kube-api-access-m5ngn\") pod \"1e84f541-2caa-40dc-9b56-f320c72f5097\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") "
Oct 30 13:36:36.323560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.322630    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-credential-socket\") pod \"64c1d503-512b-4521-af3f-668e94ed5b99\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") "
Oct 30 13:36:36.323560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.322704    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-workload-socket\") pod \"1e84f541-2caa-40dc-9b56-f320c72f5097\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") "
Oct 30 13:36:36.323560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.322791    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-workload-certs\") pod \"64c1d503-512b-4521-af3f-668e94ed5b99\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") "
Oct 30 13:36:36.323560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.322858    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c1d503-512b-4521-af3f-668e94ed5b99-kube-api-access-m964k\" (UniqueName: \"kubernetes.io/projected/64c1d503-512b-4521-af3f-668e94ed5b99-kube-api-access-m964k\") pod \"64c1d503-512b-4521-af3f-668e94ed5b99\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") "
Oct 30 13:36:36.323560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.322938    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-envoy\") pod \"1e84f541-2caa-40dc-9b56-f320c72f5097\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") "
Oct 30 13:36:36.325634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.323582    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-workload-certs" pod "64c1d503-512b-4521-af3f-668e94ed5b99" (UID: "64c1d503-512b-4521-af3f-668e94ed5b99"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.325634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.323657    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-credential-socket" pod "64c1d503-512b-4521-af3f-668e94ed5b99" (UID: "64c1d503-512b-4521-af3f-668e94ed5b99"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.325634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.324476    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-token\" (UniqueName: \"kubernetes.io/projected/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-token\") pod \"1e84f541-2caa-40dc-9b56-f320c72f5097\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") "
Oct 30 13:36:36.325634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.324547    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-data\") pod \"1e84f541-2caa-40dc-9b56-f320c72f5097\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") "
Oct 30 13:36:36.325634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.324658    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-workload-socket\") pod \"64c1d503-512b-4521-af3f-668e94ed5b99\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") "
Oct 30 13:36:36.325634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.324715    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-workload-certs\") pod \"1e84f541-2caa-40dc-9b56-f320c72f5097\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") "
Oct 30 13:36:36.325634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.324753    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-istio-data\") pod \"64c1d503-512b-4521-af3f-668e94ed5b99\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") "
Oct 30 13:36:36.325634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.324814    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-credential-socket\") pod \"1e84f541-2caa-40dc-9b56-f320c72f5097\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") "
Oct 30 13:36:36.325634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.324896    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/64c1d503-512b-4521-af3f-668e94ed5b99-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c1d503-512b-4521-af3f-668e94ed5b99-istio-podinfo\") pod \"64c1d503-512b-4521-af3f-668e94ed5b99\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") "
Oct 30 13:36:36.325634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.324934    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c1d503-512b-4521-af3f-668e94ed5b99-istio-token\" (UniqueName: \"kubernetes.io/projected/64c1d503-512b-4521-af3f-668e94ed5b99-istio-token\") pod \"64c1d503-512b-4521-af3f-668e94ed5b99\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") "
Oct 30 13:36:36.325634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.324982    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64c1d503-512b-4521-af3f-668e94ed5b99-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c1d503-512b-4521-af3f-668e94ed5b99-istiod-ca-cert\") pod \"64c1d503-512b-4521-af3f-668e94ed5b99\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") "
Oct 30 13:36:36.325634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.325189    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-podinfo\") pod \"1e84f541-2caa-40dc-9b56-f320c72f5097\" (UID: \"1e84f541-2caa-40dc-9b56-f320c72f5097\") "
Oct 30 13:36:36.325634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.325239    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-istio-envoy\") pod \"64c1d503-512b-4521-af3f-668e94ed5b99\" (UID: \"64c1d503-512b-4521-af3f-668e94ed5b99\") "
Oct 30 13:36:36.327629 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.326256    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.327629 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.326298    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.329916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.327982    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-workload-socket" pod "1e84f541-2caa-40dc-9b56-f320c72f5097" (UID: "1e84f541-2caa-40dc-9b56-f320c72f5097"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.329916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.328406    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-credential-socket" pod "1e84f541-2caa-40dc-9b56-f320c72f5097" (UID: "1e84f541-2caa-40dc-9b56-f320c72f5097"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.329916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.329301    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-data" pod "1e84f541-2caa-40dc-9b56-f320c72f5097" (UID: "1e84f541-2caa-40dc-9b56-f320c72f5097"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.329916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.329505    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-workload-socket" pod "64c1d503-512b-4521-af3f-668e94ed5b99" (UID: "64c1d503-512b-4521-af3f-668e94ed5b99"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.329916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.329717    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-workload-certs" pod "1e84f541-2caa-40dc-9b56-f320c72f5097" (UID: "1e84f541-2caa-40dc-9b56-f320c72f5097"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.329916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.329781    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e84f541-2caa-40dc-9b56-f320c72f5097-istiod-ca-cert" pod "1e84f541-2caa-40dc-9b56-f320c72f5097" (UID: "1e84f541-2caa-40dc-9b56-f320c72f5097"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:36.329916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.329896    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-istio-data" pod "64c1d503-512b-4521-af3f-668e94ed5b99" (UID: "64c1d503-512b-4521-af3f-668e94ed5b99"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.331148 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.329926    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"]
Oct 30 13:36:36.331148 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.330910    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-envoy" pod "1e84f541-2caa-40dc-9b56-f320c72f5097" (UID: "1e84f541-2caa-40dc-9b56-f320c72f5097"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.331148 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.330946    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-istio-envoy" pod "64c1d503-512b-4521-af3f-668e94ed5b99" (UID: "64c1d503-512b-4521-af3f-668e94ed5b99"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:36.331564 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.331538    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c1d503-512b-4521-af3f-668e94ed5b99-istiod-ca-cert" pod "64c1d503-512b-4521-af3f-668e94ed5b99" (UID: "64c1d503-512b-4521-af3f-668e94ed5b99"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:36.332547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.332523    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-token" pod "1e84f541-2caa-40dc-9b56-f320c72f5097" (UID: "1e84f541-2caa-40dc-9b56-f320c72f5097"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:36.334804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.334766    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c1d503-512b-4521-af3f-668e94ed5b99-istio-token" pod "64c1d503-512b-4521-af3f-668e94ed5b99" (UID: "64c1d503-512b-4521-af3f-668e94ed5b99"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:36.337663 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.337644    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e84f541-2caa-40dc-9b56-f320c72f5097-kube-api-access-m5ngn" pod "1e84f541-2caa-40dc-9b56-f320c72f5097" (UID: "1e84f541-2caa-40dc-9b56-f320c72f5097"). InnerVolumeSpecName "kube-api-access-m5ngn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:36.337748 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.337661    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c1d503-512b-4521-af3f-668e94ed5b99-kube-api-access-m964k" pod "64c1d503-512b-4521-af3f-668e94ed5b99" (UID: "64c1d503-512b-4521-af3f-668e94ed5b99"). InnerVolumeSpecName "kube-api-access-m964k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:36.337802 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.337782    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/64c1d503-512b-4521-af3f-668e94ed5b99-istio-podinfo" pod "64c1d503-512b-4521-af3f-668e94ed5b99" (UID: "64c1d503-512b-4521-af3f-668e94ed5b99"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:36:36.339042 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.339019    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-podinfo" pod "1e84f541-2caa-40dc-9b56-f320c72f5097" (UID: "1e84f541-2caa-40dc-9b56-f320c72f5097"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:36:36.340793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.340770    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"]
Oct 30 13:36:36.346227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.342799    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" containerName="istio-proxy"
Oct 30 13:36:36.349181 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.349163    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"]
Oct 30 13:36:36.349907 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.349361    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.427793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.427710    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.428042 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.427834    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.428042 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.427857    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.428042 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.427891    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.428248 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428157    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.428339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428318    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.428401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428360    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.428488 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428454    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjk5\" (UniqueName: \"kubernetes.io/projected/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-kube-api-access-ggjk5\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.428623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428607    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.428723 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428712    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m964k\" (UniqueName: \"kubernetes.io/projected/64c1d503-512b-4521-af3f-668e94ed5b99-kube-api-access-m964k\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.428784 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428752    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.428784 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428772    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.428901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428787    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.428901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428800    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.428901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428816    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.428901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428830    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.428901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428843    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.428901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428857    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/64c1d503-512b-4521-af3f-668e94ed5b99-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.428901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428871    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/64c1d503-512b-4521-af3f-668e94ed5b99-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.428901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428884    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/64c1d503-512b-4521-af3f-668e94ed5b99-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.428901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428898    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1e84f541-2caa-40dc-9b56-f320c72f5097-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.429254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428910    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/64c1d503-512b-4521-af3f-668e94ed5b99-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.429254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428924    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1e84f541-2caa-40dc-9b56-f320c72f5097-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.429254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428938    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5ngn\" (UniqueName: \"kubernetes.io/projected/1e84f541-2caa-40dc-9b56-f320c72f5097-kube-api-access-m5ngn\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.429254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.428953    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1e84f541-2caa-40dc-9b56-f320c72f5097-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:36:36.530021 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.529989    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.530234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530031    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.530234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530048    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.530234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530116    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.530234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530158    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.530234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530173    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.530234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530223    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj9j9\" (UniqueName: \"kubernetes.io/projected/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-kube-api-access-dj9j9\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530302    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530320    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530343    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530351    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530474    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530502    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530527    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530553    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530596    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530700    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530713    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530748    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjk5\" (UniqueName: \"kubernetes.io/projected/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-kube-api-access-ggjk5\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530779    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530857    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.530888    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.531486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.531462    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.534092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.534069    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.534200 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.534171    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.556652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.556312    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.557965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.557939    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjk5\" (UniqueName: \"kubernetes.io/projected/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-kube-api-access-ggjk5\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.632989 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.632950    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.633190 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.633094    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.633331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.633310    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.633416 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.633364    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.633491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.633422    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.633491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.633446    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.633619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.633599    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-workload-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.633686 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.633656    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj9j9\" (UniqueName: \"kubernetes.io/projected/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-kube-api-access-dj9j9\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.633778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.633757    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-workload-certs\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.634713 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.633810    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-data\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.634713 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.633914    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.634713 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.633956    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.634713 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.634421    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-credential-socket\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.635313 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.634834    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istiod-ca-cert\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.635808 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.635792    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-envoy\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.636147 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.636133    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-podinfo\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.640215 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.640194    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:36.655453 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.655433    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-token\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.655685 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.655661    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj9j9\" (UniqueName: \"kubernetes.io/projected/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-kube-api-access-dj9j9\") pod \"e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") " pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.672603 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.672571    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:36.855303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.854669    5893 generic.go:356] "Generic (PLEG): container finished" podID="1e84f541-2caa-40dc-9b56-f320c72f5097" containerID="34e990fe417fe794072cafeb38ede2453d16da8d8f8b390fe9735a5adb817e00" exitCode=0
Oct 30 13:36:36.855303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.854868    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz" event={"ID":"1e84f541-2caa-40dc-9b56-f320c72f5097","Type":"ContainerDied","Data":"34e990fe417fe794072cafeb38ede2453d16da8d8f8b390fe9735a5adb817e00"}
Oct 30 13:36:36.855303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.854924    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz" event={"ID":"1e84f541-2caa-40dc-9b56-f320c72f5097","Type":"ContainerDied","Data":"f59a8ee7d5e1591aeaaab932ceb890fd698facc0f21eb0d9c57d1f2a1e395f12"}
Oct 30 13:36:36.855303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.854957    5893 scope.go:122] "RemoveContainer" containerID="34e990fe417fe794072cafeb38ede2453d16da8d8f8b390fe9735a5adb817e00"
Oct 30 13:36:36.855303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.855122    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"
Oct 30 13:36:36.860105 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.860074    5893 generic.go:356] "Generic (PLEG): container finished" podID="64c1d503-512b-4521-af3f-668e94ed5b99" containerID="47ebcee4149f5122d1f56ec827bf0551e474652c23fa8a02a5d075f0a55c8dcd" exitCode=0
Oct 30 13:36:36.860337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.860312    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" event={"ID":"64c1d503-512b-4521-af3f-668e94ed5b99","Type":"ContainerDied","Data":"47ebcee4149f5122d1f56ec827bf0551e474652c23fa8a02a5d075f0a55c8dcd"}
Oct 30 13:36:36.860472 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.860366    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc" event={"ID":"64c1d503-512b-4521-af3f-668e94ed5b99","Type":"ContainerDied","Data":"68389413d0431cdcf850fa9b7bb3c2f2d249e2b647c0e51631aa6702e250ca53"}
Oct 30 13:36:36.860472 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.860427    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"
Oct 30 13:36:36.892858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.892832    5893 scope.go:122] "RemoveContainer" containerID="34e990fe417fe794072cafeb38ede2453d16da8d8f8b390fe9735a5adb817e00"
Oct 30 13:36:36.893214 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:36:36.893189    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34e990fe417fe794072cafeb38ede2453d16da8d8f8b390fe9735a5adb817e00\": container with ID starting with 34e990fe417fe794072cafeb38ede2453d16da8d8f8b390fe9735a5adb817e00 not found: ID does not exist" containerID="34e990fe417fe794072cafeb38ede2453d16da8d8f8b390fe9735a5adb817e00"
Oct 30 13:36:36.893361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.893225    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e990fe417fe794072cafeb38ede2453d16da8d8f8b390fe9735a5adb817e00"} err="failed to get container status \"34e990fe417fe794072cafeb38ede2453d16da8d8f8b390fe9735a5adb817e00\": rpc error: code = NotFound desc = could not find container \"34e990fe417fe794072cafeb38ede2453d16da8d8f8b390fe9735a5adb817e00\": container with ID starting with 34e990fe417fe794072cafeb38ede2453d16da8d8f8b390fe9735a5adb817e00 not found: ID does not exist"
Oct 30 13:36:36.893361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.893243    5893 scope.go:122] "RemoveContainer" containerID="47ebcee4149f5122d1f56ec827bf0551e474652c23fa8a02a5d075f0a55c8dcd"
Oct 30 13:36:36.906741 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.906062    5893 scope.go:122] "RemoveContainer" containerID="214caddcae79781a68b6918a19ba6a299bf15c42a3f346f857ba5868515234e5"
Oct 30 13:36:36.918689 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.918666    5893 scope.go:122] "RemoveContainer" containerID="47ebcee4149f5122d1f56ec827bf0551e474652c23fa8a02a5d075f0a55c8dcd"
Oct 30 13:36:36.919317 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:36:36.919024    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47ebcee4149f5122d1f56ec827bf0551e474652c23fa8a02a5d075f0a55c8dcd\": container with ID starting with 47ebcee4149f5122d1f56ec827bf0551e474652c23fa8a02a5d075f0a55c8dcd not found: ID does not exist" containerID="47ebcee4149f5122d1f56ec827bf0551e474652c23fa8a02a5d075f0a55c8dcd"
Oct 30 13:36:36.919317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.919053    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47ebcee4149f5122d1f56ec827bf0551e474652c23fa8a02a5d075f0a55c8dcd"} err="failed to get container status \"47ebcee4149f5122d1f56ec827bf0551e474652c23fa8a02a5d075f0a55c8dcd\": rpc error: code = NotFound desc = could not find container \"47ebcee4149f5122d1f56ec827bf0551e474652c23fa8a02a5d075f0a55c8dcd\": container with ID starting with 47ebcee4149f5122d1f56ec827bf0551e474652c23fa8a02a5d075f0a55c8dcd not found: ID does not exist"
Oct 30 13:36:36.919317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.919073    5893 scope.go:122] "RemoveContainer" containerID="214caddcae79781a68b6918a19ba6a299bf15c42a3f346f857ba5868515234e5"
Oct 30 13:36:36.920305 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:36:36.919412    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"214caddcae79781a68b6918a19ba6a299bf15c42a3f346f857ba5868515234e5\": container with ID starting with 214caddcae79781a68b6918a19ba6a299bf15c42a3f346f857ba5868515234e5 not found: ID does not exist" containerID="214caddcae79781a68b6918a19ba6a299bf15c42a3f346f857ba5868515234e5"
Oct 30 13:36:36.920305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.919445    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"214caddcae79781a68b6918a19ba6a299bf15c42a3f346f857ba5868515234e5"} err="failed to get container status \"214caddcae79781a68b6918a19ba6a299bf15c42a3f346f857ba5868515234e5\": rpc error: code = NotFound desc = could not find container \"214caddcae79781a68b6918a19ba6a299bf15c42a3f346f857ba5868515234e5\": container with ID starting with 214caddcae79781a68b6918a19ba6a299bf15c42a3f346f857ba5868515234e5 not found: ID does not exist"
Oct 30 13:36:36.923053 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.923026    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"]
Oct 30 13:36:36.927224 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.927194    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-rmcpz"]
Oct 30 13:36:36.962212 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.962120    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"]
Oct 30 13:36:36.971974 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.971948    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-1ba901-openshift-default-5c5d49d965-m2fnc"]
Oct 30 13:36:36.983969 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.983940    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"]
Oct 30 13:36:36.988101 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:36:36.988075    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e678e9e_ec06_4b9e_b1f6_cebb61e59571.slice/crio-a712adca07201779a0f01a54e681474eba632d7258ca8b33c5b3034949e6a4be WatchSource:0}: Error finding container a712adca07201779a0f01a54e681474eba632d7258ca8b33c5b3034949e6a4be: Status 404 returned error can't find the container with id a712adca07201779a0f01a54e681474eba632d7258ca8b33c5b3034949e6a4be
Oct 30 13:36:36.991433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.991396    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:36:36.991533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.991475    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:36:36.991533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:36.991506    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:36:37.027885 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:37.027846    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"]
Oct 30 13:36:37.028673 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:36:37.028645    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod895b7a55_b271_4b61_a3c5_7411e8b9a9b0.slice/crio-d9ef5342e6b0205ff79465db0380407d5fecc1610fe5a18cb084f5bed3b2979b WatchSource:0}: Error finding container d9ef5342e6b0205ff79465db0380407d5fecc1610fe5a18cb084f5bed3b2979b: Status 404 returned error can't find the container with id d9ef5342e6b0205ff79465db0380407d5fecc1610fe5a18cb084f5bed3b2979b
Oct 30 13:36:37.032010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:37.031981    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:36:37.032090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:37.032049    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:36:37.032090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:37.032078    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:36:37.923224 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:37.923186    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh" event={"ID":"895b7a55-b271-4b61-a3c5-7411e8b9a9b0","Type":"ContainerStarted","Data":"87554b331c25f5c68c62eadcb8571a70cc0c5c059b6a8a250c0baee48a992e57"}
Oct 30 13:36:37.923665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:37.923247    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh" event={"ID":"895b7a55-b271-4b61-a3c5-7411e8b9a9b0","Type":"ContainerStarted","Data":"d9ef5342e6b0205ff79465db0380407d5fecc1610fe5a18cb084f5bed3b2979b"}
Oct 30 13:36:37.926712 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:37.926689    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg" event={"ID":"5e678e9e-ec06-4b9e-b1f6-cebb61e59571","Type":"ContainerStarted","Data":"d1deecbc92d7224dc05cdb0e0940518baa5171fb41841eae190fe62cb1cf5553"}
Oct 30 13:36:37.926824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:37.926732    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg" event={"ID":"5e678e9e-ec06-4b9e-b1f6-cebb61e59571","Type":"ContainerStarted","Data":"a712adca07201779a0f01a54e681474eba632d7258ca8b33c5b3034949e6a4be"}
Oct 30 13:36:38.002300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:37.969984    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh" podStartSLOduration=60.969962187 podStartE2EDuration="1m0.969962187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:37.967729569 +0000 UTC m=+3039.551159261" watchObservedRunningTime="2026-10-30 13:36:37.969962187 +0000 UTC m=+3039.553391881"
Oct 30 13:36:38.080141 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:38.068455    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg" podStartSLOduration=16.068430508 podStartE2EDuration="16.068430508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:38.057997213 +0000 UTC m=+3039.641426906" watchObservedRunningTime="2026-10-30 13:36:38.068430508 +0000 UTC m=+3039.651860201"
Oct 30 13:36:38.535524 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:38.535493    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e84f541-2caa-40dc-9b56-f320c72f5097" path="/var/lib/kubelet/pods/1e84f541-2caa-40dc-9b56-f320c72f5097/volumes"
Oct 30 13:36:38.536246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:38.536222    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64c1d503-512b-4521-af3f-668e94ed5b99" path="/var/lib/kubelet/pods/64c1d503-512b-4521-af3f-668e94ed5b99/volumes"
Oct 30 13:36:38.654298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:38.646724    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:38.654298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:38.646858    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 13:36:38.654298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:38.646901    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg" podUID="5e678e9e-ec06-4b9e-b1f6-cebb61e59571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 13:36:38.685302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:38.682322    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:38.685302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:38.682452    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 13:36:38.685302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:38.682494    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh" podUID="895b7a55-b271-4b61-a3c5-7411e8b9a9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 13:36:39.648729 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:39.648689    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 13:36:39.649355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:39.649320    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg" podUID="5e678e9e-ec06-4b9e-b1f6-cebb61e59571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 13:36:39.675813 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:39.675777    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 13:36:39.676090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:39.676059    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh" podUID="895b7a55-b271-4b61-a3c5-7411e8b9a9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 13:36:40.648299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:40.644653    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 13:36:40.648299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:40.644716    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg" podUID="5e678e9e-ec06-4b9e-b1f6-cebb61e59571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 13:36:40.678297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:40.677641    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 13:36:40.678297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:40.677699    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh" podUID="895b7a55-b271-4b61-a3c5-7411e8b9a9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 13:36:41.646697 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:41.646662    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 13:36:41.651296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:41.646966    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg" podUID="5e678e9e-ec06-4b9e-b1f6-cebb61e59571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 13:36:41.687648 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:41.687612    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 13:36:41.688240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:41.688212    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh" podUID="895b7a55-b271-4b61-a3c5-7411e8b9a9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 13:36:41.910792 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:41.910712    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"]
Oct 30 13:36:42.020693 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:42.020659    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"]
Oct 30 13:36:42.658306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:42.656402    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 13:36:42.658306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:42.656464    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg" podUID="5e678e9e-ec06-4b9e-b1f6-cebb61e59571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 13:36:42.684301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:42.679188    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 13:36:42.684301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:42.679247    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh" podUID="895b7a55-b271-4b61-a3c5-7411e8b9a9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 13:36:43.642180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:43.642139    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Oct 30 13:36:43.643172 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:43.642205    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg" podUID="5e678e9e-ec06-4b9e-b1f6-cebb61e59571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Oct 30 13:36:43.674840 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:43.674798    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused" start-of-body=
Oct 30 13:36:43.675044 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:43.674879    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh" podUID="895b7a55-b271-4b61-a3c5-7411e8b9a9b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.200:15021/healthz/ready\": dial tcp 10.134.0.200:15021: connect: connection refused"
Oct 30 13:36:44.655822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:44.654665    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:44.656549 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:44.656524    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:44.662025 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:44.658815    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:36:44.679196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:44.679167    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:44.679538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:44.679515    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:44.680880 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:44.680858    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:36:46.911798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:46.911739    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn" podUID="d445892f-5fd7-42a2-ae10-f779e2e37dfa" containerName="payload-processing" containerID="cri-o://08f81ac6824bc23ab0284348259605095988dc77806772bb3d3a99ea6699b385" gracePeriod=25
Oct 30 13:36:47.021061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:47.021025    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p" podUID="2b81d985-3126-4e47-b9f9-6e9665d7dff0" containerName="payload-pre-processing" containerID="cri-o://d1d68779b685da760413e427b6aeb775889d18c1b94b01a8d67180f074617228" gracePeriod=25
Oct 30 13:36:48.650760 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:36:48.650715    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:36:54.582447 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:54.582413    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn" podUID="d445892f-5fd7-42a2-ae10-f779e2e37dfa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:36:54.582447 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:36:54.582438    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p" podUID="2b81d985-3126-4e47-b9f9-6e9665d7dff0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:37:04.582683 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:04.582642    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p" podUID="2b81d985-3126-4e47-b9f9-6e9665d7dff0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:37:04.582683 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:04.582666    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn" podUID="d445892f-5fd7-42a2-ae10-f779e2e37dfa" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:37:07.256351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.254682    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"
Oct 30 13:37:07.319951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.319922    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d445892f-5fd7-42a2-ae10-f779e2e37dfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d445892f-5fd7-42a2-ae10-f779e2e37dfa-plugins-config-volume\") pod \"d445892f-5fd7-42a2-ae10-f779e2e37dfa\" (UID: \"d445892f-5fd7-42a2-ae10-f779e2e37dfa\") "
Oct 30 13:37:07.320120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.319988    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d445892f-5fd7-42a2-ae10-f779e2e37dfa-kube-api-access-znj2t\" (UniqueName: \"kubernetes.io/projected/d445892f-5fd7-42a2-ae10-f779e2e37dfa-kube-api-access-znj2t\") pod \"d445892f-5fd7-42a2-ae10-f779e2e37dfa\" (UID: \"d445892f-5fd7-42a2-ae10-f779e2e37dfa\") "
Oct 30 13:37:07.322897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.322864    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d445892f-5fd7-42a2-ae10-f779e2e37dfa-plugins-config-volume" pod "d445892f-5fd7-42a2-ae10-f779e2e37dfa" (UID: "d445892f-5fd7-42a2-ae10-f779e2e37dfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:07.329857 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.329822    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d445892f-5fd7-42a2-ae10-f779e2e37dfa-kube-api-access-znj2t" pod "d445892f-5fd7-42a2-ae10-f779e2e37dfa" (UID: "d445892f-5fd7-42a2-ae10-f779e2e37dfa"). InnerVolumeSpecName "kube-api-access-znj2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:07.330015 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.329941    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"]
Oct 30 13:37:07.331448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.331421    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d445892f-5fd7-42a2-ae10-f779e2e37dfa" containerName="payload-processing"
Oct 30 13:37:07.331448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.331448    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d445892f-5fd7-42a2-ae10-f779e2e37dfa" containerName="payload-processing"
Oct 30 13:37:07.332221 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.331768    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d445892f-5fd7-42a2-ae10-f779e2e37dfa" containerName="payload-processing"
Oct 30 13:37:07.357851 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.357818    5893 generic.go:356] "Generic (PLEG): container finished" podID="d445892f-5fd7-42a2-ae10-f779e2e37dfa" containerID="08f81ac6824bc23ab0284348259605095988dc77806772bb3d3a99ea6699b385" exitCode=0
Oct 30 13:37:07.358048 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.358001    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"
Oct 30 13:37:07.371827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.371799    5893 generic.go:356] "Generic (PLEG): container finished" podID="2b81d985-3126-4e47-b9f9-6e9665d7dff0" containerID="d1d68779b685da760413e427b6aeb775889d18c1b94b01a8d67180f074617228" exitCode=0
Oct 30 13:37:07.373489 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.373464    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn" event={"ID":"d445892f-5fd7-42a2-ae10-f779e2e37dfa","Type":"ContainerDied","Data":"08f81ac6824bc23ab0284348259605095988dc77806772bb3d3a99ea6699b385"}
Oct 30 13:37:07.373656 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.373639    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn" event={"ID":"d445892f-5fd7-42a2-ae10-f779e2e37dfa","Type":"ContainerDied","Data":"2d9a77b67cec72bd93422f84237c1de09f4018716ee77ceab0746cdd2296728d"}
Oct 30 13:37:07.373751 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.373738    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p" event={"ID":"2b81d985-3126-4e47-b9f9-6e9665d7dff0","Type":"ContainerDied","Data":"d1d68779b685da760413e427b6aeb775889d18c1b94b01a8d67180f074617228"}
Oct 30 13:37:07.373873 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.373851    5893 scope.go:122] "RemoveContainer" containerID="08f81ac6824bc23ab0284348259605095988dc77806772bb3d3a99ea6699b385"
Oct 30 13:37:07.373962 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.373768    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"
Oct 30 13:37:07.380812 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.380787    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"]
Oct 30 13:37:07.390921 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.390901    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c91909-dockercfg-n7thm\""
Oct 30 13:37:07.391355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.391330    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c91909\""
Oct 30 13:37:07.400364 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.400340    5893 scope.go:122] "RemoveContainer" containerID="08f81ac6824bc23ab0284348259605095988dc77806772bb3d3a99ea6699b385"
Oct 30 13:37:07.400729 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:37:07.400705    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f81ac6824bc23ab0284348259605095988dc77806772bb3d3a99ea6699b385\": container with ID starting with 08f81ac6824bc23ab0284348259605095988dc77806772bb3d3a99ea6699b385 not found: ID does not exist" containerID="08f81ac6824bc23ab0284348259605095988dc77806772bb3d3a99ea6699b385"
Oct 30 13:37:07.400824 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.400743    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f81ac6824bc23ab0284348259605095988dc77806772bb3d3a99ea6699b385"} err="failed to get container status \"08f81ac6824bc23ab0284348259605095988dc77806772bb3d3a99ea6699b385\": rpc error: code = NotFound desc = could not find container \"08f81ac6824bc23ab0284348259605095988dc77806772bb3d3a99ea6699b385\": container with ID starting with 08f81ac6824bc23ab0284348259605095988dc77806772bb3d3a99ea6699b385 not found: ID does not exist"
Oct 30 13:37:07.410502 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.410475    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"
Oct 30 13:37:07.423351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.423322    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bd8ee2-cb78-4333-bccc-e1c7b34286c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj\" (UID: \"21bd8ee2-cb78-4333-bccc-e1c7b34286c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"
Oct 30 13:37:07.423576 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.423560    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j96p\" (UniqueName: \"kubernetes.io/projected/21bd8ee2-cb78-4333-bccc-e1c7b34286c7-kube-api-access-2j96p\") pod \"payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj\" (UID: \"21bd8ee2-cb78-4333-bccc-e1c7b34286c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"
Oct 30 13:37:07.423735 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.423718    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d445892f-5fd7-42a2-ae10-f779e2e37dfa-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:07.423789 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.423740    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znj2t\" (UniqueName: \"kubernetes.io/projected/d445892f-5fd7-42a2-ae10-f779e2e37dfa-kube-api-access-znj2t\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:07.461400 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.461308    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"]
Oct 30 13:37:07.466451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.466425    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-03b5b3-6fb5db6d74-c7wrn"]
Oct 30 13:37:07.524484 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.524450    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b81d985-3126-4e47-b9f9-6e9665d7dff0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b81d985-3126-4e47-b9f9-6e9665d7dff0-plugins-config-volume\") pod \"2b81d985-3126-4e47-b9f9-6e9665d7dff0\" (UID: \"2b81d985-3126-4e47-b9f9-6e9665d7dff0\") "
Oct 30 13:37:07.524699 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.524674    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b81d985-3126-4e47-b9f9-6e9665d7dff0-kube-api-access-8x7lr\" (UniqueName: \"kubernetes.io/projected/2b81d985-3126-4e47-b9f9-6e9665d7dff0-kube-api-access-8x7lr\") pod \"2b81d985-3126-4e47-b9f9-6e9665d7dff0\" (UID: \"2b81d985-3126-4e47-b9f9-6e9665d7dff0\") "
Oct 30 13:37:07.524954 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.524927    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b81d985-3126-4e47-b9f9-6e9665d7dff0-plugins-config-volume" pod "2b81d985-3126-4e47-b9f9-6e9665d7dff0" (UID: "2b81d985-3126-4e47-b9f9-6e9665d7dff0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:07.525345 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.525315    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j96p\" (UniqueName: \"kubernetes.io/projected/21bd8ee2-cb78-4333-bccc-e1c7b34286c7-kube-api-access-2j96p\") pod \"payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj\" (UID: \"21bd8ee2-cb78-4333-bccc-e1c7b34286c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"
Oct 30 13:37:07.525615 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.525593    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bd8ee2-cb78-4333-bccc-e1c7b34286c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj\" (UID: \"21bd8ee2-cb78-4333-bccc-e1c7b34286c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"
Oct 30 13:37:07.526118 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.525874    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b81d985-3126-4e47-b9f9-6e9665d7dff0-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:07.526752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.526730    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bd8ee2-cb78-4333-bccc-e1c7b34286c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj\" (UID: \"21bd8ee2-cb78-4333-bccc-e1c7b34286c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"
Oct 30 13:37:07.527673 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.527648    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b81d985-3126-4e47-b9f9-6e9665d7dff0-kube-api-access-8x7lr" pod "2b81d985-3126-4e47-b9f9-6e9665d7dff0" (UID: "2b81d985-3126-4e47-b9f9-6e9665d7dff0"). InnerVolumeSpecName "kube-api-access-8x7lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:07.554133 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.554098    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j96p\" (UniqueName: \"kubernetes.io/projected/21bd8ee2-cb78-4333-bccc-e1c7b34286c7-kube-api-access-2j96p\") pod \"payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj\" (UID: \"21bd8ee2-cb78-4333-bccc-e1c7b34286c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"
Oct 30 13:37:07.627399 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.627372    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x7lr\" (UniqueName: \"kubernetes.io/projected/2b81d985-3126-4e47-b9f9-6e9665d7dff0-kube-api-access-8x7lr\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:07.703264 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:07.703235    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"
Oct 30 13:37:08.021165 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:08.021137    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"]
Oct 30 13:37:08.024107 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:37:08.024067    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21bd8ee2_cb78_4333_bccc_e1c7b34286c7.slice/crio-e7b09449f98bd61a9d3795a9506a92ef41fe92f83371764ab596b86c03c539ff WatchSource:0}: Error finding container e7b09449f98bd61a9d3795a9506a92ef41fe92f83371764ab596b86c03c539ff: Status 404 returned error can't find the container with id e7b09449f98bd61a9d3795a9506a92ef41fe92f83371764ab596b86c03c539ff
Oct 30 13:37:08.386796 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:08.386706    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p" event={"ID":"2b81d985-3126-4e47-b9f9-6e9665d7dff0","Type":"ContainerDied","Data":"50f498ad5a87e6449471e00715acba683993d20d9f2e9d906240d34eeb169fdf"}
Oct 30 13:37:08.387324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:08.387298    5893 scope.go:122] "RemoveContainer" containerID="d1d68779b685da760413e427b6aeb775889d18c1b94b01a8d67180f074617228"
Oct 30 13:37:08.387324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:08.386719    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"
Oct 30 13:37:08.388874 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:08.388822    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj" event={"ID":"21bd8ee2-cb78-4333-bccc-e1c7b34286c7","Type":"ContainerStarted","Data":"4e3f93b8d5d6cdceac21b5777955ad85c5a4377c93d5c4c72d25e1074e234db6"}
Oct 30 13:37:08.388967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:08.388883    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj" event={"ID":"21bd8ee2-cb78-4333-bccc-e1c7b34286c7","Type":"ContainerStarted","Data":"e7b09449f98bd61a9d3795a9506a92ef41fe92f83371764ab596b86c03c539ff"}
Oct 30 13:37:08.389032 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:08.389015    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"
Oct 30 13:37:08.443062 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:08.443007    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj" podStartSLOduration=153.442988462 podStartE2EDuration="2m33.442988462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:08.438524666 +0000 UTC m=+3070.021954363" watchObservedRunningTime="2026-10-30 13:37:08.442988462 +0000 UTC m=+3070.026418156"
Oct 30 13:37:08.476998 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:08.476936    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"]
Oct 30 13:37:08.481506 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:08.481479    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-03b5b3-7fbc5f64c-8r86p"]
Oct 30 13:37:08.534058 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:08.533982    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b81d985-3126-4e47-b9f9-6e9665d7dff0" path="/var/lib/kubelet/pods/2b81d985-3126-4e47-b9f9-6e9665d7dff0/volumes"
Oct 30 13:37:08.534542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:08.534526    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d445892f-5fd7-42a2-ae10-f779e2e37dfa" path="/var/lib/kubelet/pods/d445892f-5fd7-42a2-ae10-f779e2e37dfa/volumes"
Oct 30 13:37:18.648184 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:37:18.648143    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:37:19.402010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:19.401981    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"
Oct 30 13:37:19.572777 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:19.572743    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"]
Oct 30 13:37:19.573024 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:19.572995    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn" podUID="91c3dae9-4165-4660-bff5-f728025903cd" containerName="istio-proxy" containerID="cri-o://c1a79b3af28d0b470e7871b0474881f3c3512f5d3adba73c351966fd473f7504" gracePeriod=30
Oct 30 13:37:19.805729 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:19.805698    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"]
Oct 30 13:37:19.806237 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:19.805947    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz" podUID="c0bfdfdc-1f43-4127-b050-4e8a2c2f516f" containerName="maas-api" containerID="cri-o://9ae9def9cd2a2b672780ae0fbf1d1f1317099ebf2adc4cca1b2ff900069eece4" gracePeriod=30
Oct 30 13:37:19.988427 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:19.988394    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"]
Oct 30 13:37:19.988731 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:19.988704    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl" podUID="01f1034a-cc31-43bc-91d4-8c0fc67e5960" containerName="payload-processing" containerID="cri-o://d16a7c588fa50f4f52ed027cda22c53cd17b743257f57419b7223e2853897648" gracePeriod=30
Oct 30 13:37:20.023662 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.023392    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"]
Oct 30 13:37:20.023662 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.023624    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n" podUID="d3f79cd8-b784-4ef6-9a74-69a815c6e8a8" containerName="payload-pre-processing" containerID="cri-o://707286ec786736b8b5f55628f6379022f1ad10a0f654c8dd6c6462cbad4addce" gracePeriod=30
Oct 30 13:37:20.167361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.166846    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:37:20.224997 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.224963    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"]
Oct 30 13:37:20.225859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.225836    5893 cpu_manager.go:414] "containerMap: removing container" podUID="c0bfdfdc-1f43-4127-b050-4e8a2c2f516f" containerName="maas-api"
Oct 30 13:37:20.225859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.225856    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0bfdfdc-1f43-4127-b050-4e8a2c2f516f" containerName="maas-api"
Oct 30 13:37:20.226075 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.225877    5893 cpu_manager.go:414] "containerMap: removing container" podUID="2b81d985-3126-4e47-b9f9-6e9665d7dff0" containerName="payload-pre-processing"
Oct 30 13:37:20.226075 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.225883    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b81d985-3126-4e47-b9f9-6e9665d7dff0" containerName="payload-pre-processing"
Oct 30 13:37:20.226075 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.226025    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b81d985-3126-4e47-b9f9-6e9665d7dff0" containerName="payload-pre-processing"
Oct 30 13:37:20.226075 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.226046    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0bfdfdc-1f43-4127-b050-4e8a2c2f516f" containerName="maas-api"
Oct 30 13:37:20.230143 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.230119    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.245326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.243429    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-ba8145-openshift-default-dockercfg-gtq7f\""
Oct 30 13:37:20.264619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.264336    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-openshift-service-ca\") pod \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") "
Oct 30 13:37:20.264619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.264515    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-kube-api-access-24pxr\" (UniqueName: \"kubernetes.io/projected/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-kube-api-access-24pxr\") pod \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") "
Oct 30 13:37:20.264858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.264748    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-metrics-certs\") pod \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") "
Oct 30 13:37:20.264858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.264826    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-maas-api-tls\") pod \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\" (UID: \"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f\") "
Oct 30 13:37:20.266248 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.265654    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-openshift-service-ca" pod "c0bfdfdc-1f43-4127-b050-4e8a2c2f516f" (UID: "c0bfdfdc-1f43-4127-b050-4e8a2c2f516f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:20.266248 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.265791    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:20.266568 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.266544    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"]
Oct 30 13:37:20.269171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.269098    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-maas-api-tls" pod "c0bfdfdc-1f43-4127-b050-4e8a2c2f516f" (UID: "c0bfdfdc-1f43-4127-b050-4e8a2c2f516f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:37:20.284478 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.284444    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-kube-api-access-24pxr" pod "c0bfdfdc-1f43-4127-b050-4e8a2c2f516f" (UID: "c0bfdfdc-1f43-4127-b050-4e8a2c2f516f"). InnerVolumeSpecName "kube-api-access-24pxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:20.285206 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.285168    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-metrics-certs" pod "c0bfdfdc-1f43-4127-b050-4e8a2c2f516f" (UID: "c0bfdfdc-1f43-4127-b050-4e8a2c2f516f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:37:20.525303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.498491    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-workload-socket\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.525303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.498718    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-token\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.525303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.498761    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-data\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.525303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.498828    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-credential-socket\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.525303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.498863    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e02b51d-4bfe-4805-b52e-d65419291f1e-istiod-ca-cert\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.525303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.499112    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-envoy\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.525303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.499155    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt5zj\" (UniqueName: \"kubernetes.io/projected/6e02b51d-4bfe-4805-b52e-d65419291f1e-kube-api-access-bt5zj\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.525303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.499205    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-workload-certs\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.525303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.499303    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-podinfo\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.525303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.499457    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24pxr\" (UniqueName: \"kubernetes.io/projected/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-kube-api-access-24pxr\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:20.525303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.499477    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:20.525303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.499492    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:20.720304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.720245    5893 generic.go:356] "Generic (PLEG): container finished" podID="01f1034a-cc31-43bc-91d4-8c0fc67e5960" containerID="d16a7c588fa50f4f52ed027cda22c53cd17b743257f57419b7223e2853897648" exitCode=0
Oct 30 13:37:20.722375 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.722353    5893 generic.go:356] "Generic (PLEG): container finished" podID="c0bfdfdc-1f43-4127-b050-4e8a2c2f516f" containerID="9ae9def9cd2a2b672780ae0fbf1d1f1317099ebf2adc4cca1b2ff900069eece4" exitCode=0
Oct 30 13:37:20.722684 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.722640    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl" event={"ID":"01f1034a-cc31-43bc-91d4-8c0fc67e5960","Type":"ContainerDied","Data":"d16a7c588fa50f4f52ed027cda22c53cd17b743257f57419b7223e2853897648"}
Oct 30 13:37:20.722755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.722697    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz" event={"ID":"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f","Type":"ContainerDied","Data":"9ae9def9cd2a2b672780ae0fbf1d1f1317099ebf2adc4cca1b2ff900069eece4"}
Oct 30 13:37:20.722755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.722718    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz" event={"ID":"c0bfdfdc-1f43-4127-b050-4e8a2c2f516f","Type":"ContainerDied","Data":"d0988a180ec2596f291666616029f49ef77e8580884e4ff28617370b611f440a"}
Oct 30 13:37:20.722755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.722739    5893 scope.go:122] "RemoveContainer" containerID="9ae9def9cd2a2b672780ae0fbf1d1f1317099ebf2adc4cca1b2ff900069eece4"
Oct 30 13:37:20.723791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.723761    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"
Oct 30 13:37:20.724129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.724112    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"
Oct 30 13:37:20.724584 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.724560    5893 generic.go:356] "Generic (PLEG): container finished" podID="d3f79cd8-b784-4ef6-9a74-69a815c6e8a8" containerID="707286ec786736b8b5f55628f6379022f1ad10a0f654c8dd6c6462cbad4addce" exitCode=0
Oct 30 13:37:20.724694 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.724609    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n" event={"ID":"d3f79cd8-b784-4ef6-9a74-69a815c6e8a8","Type":"ContainerDied","Data":"707286ec786736b8b5f55628f6379022f1ad10a0f654c8dd6c6462cbad4addce"}
Oct 30 13:37:20.727682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.727659    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"
Oct 30 13:37:20.738009 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.737979    5893 scope.go:122] "RemoveContainer" containerID="9ae9def9cd2a2b672780ae0fbf1d1f1317099ebf2adc4cca1b2ff900069eece4"
Oct 30 13:37:20.738365 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:37:20.738338    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae9def9cd2a2b672780ae0fbf1d1f1317099ebf2adc4cca1b2ff900069eece4\": container with ID starting with 9ae9def9cd2a2b672780ae0fbf1d1f1317099ebf2adc4cca1b2ff900069eece4 not found: ID does not exist" containerID="9ae9def9cd2a2b672780ae0fbf1d1f1317099ebf2adc4cca1b2ff900069eece4"
Oct 30 13:37:20.738472 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.738377    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae9def9cd2a2b672780ae0fbf1d1f1317099ebf2adc4cca1b2ff900069eece4"} err="failed to get container status \"9ae9def9cd2a2b672780ae0fbf1d1f1317099ebf2adc4cca1b2ff900069eece4\": rpc error: code = NotFound desc = could not find container \"9ae9def9cd2a2b672780ae0fbf1d1f1317099ebf2adc4cca1b2ff900069eece4\": container with ID starting with 9ae9def9cd2a2b672780ae0fbf1d1f1317099ebf2adc4cca1b2ff900069eece4 not found: ID does not exist"
Oct 30 13:37:20.768601 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.768568    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-token\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.768831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.768812    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-data\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.769006 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.768989    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-credential-socket\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.769137 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.769121    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e02b51d-4bfe-4805-b52e-d65419291f1e-istiod-ca-cert\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.769481 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.769462    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-envoy\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.770044 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.770021    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-credential-socket\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.772109 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.770149    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5zj\" (UniqueName: \"kubernetes.io/projected/6e02b51d-4bfe-4805-b52e-d65419291f1e-kube-api-access-bt5zj\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.772326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.772306    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-workload-certs\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.772527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.772509    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-podinfo\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.772652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.772635    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-workload-socket\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.773181 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.773163    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-workload-socket\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.774313 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.773705    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-workload-certs\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.774313 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.771088    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-data\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.774313 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.772019    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e02b51d-4bfe-4805-b52e-d65419291f1e-istiod-ca-cert\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.774505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.774223    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-envoy\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.776650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.776597    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"]
Oct 30 13:37:20.778156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.778135    5893 cpu_manager.go:414] "containerMap: removing container" podUID="01f1034a-cc31-43bc-91d4-8c0fc67e5960" containerName="payload-processing"
Oct 30 13:37:20.778358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.778158    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01f1034a-cc31-43bc-91d4-8c0fc67e5960" containerName="payload-processing"
Oct 30 13:37:20.778358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.778223    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d3f79cd8-b784-4ef6-9a74-69a815c6e8a8" containerName="payload-pre-processing"
Oct 30 13:37:20.778358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.778233    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f79cd8-b784-4ef6-9a74-69a815c6e8a8" containerName="payload-pre-processing"
Oct 30 13:37:20.778586 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.778487    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="01f1034a-cc31-43bc-91d4-8c0fc67e5960" containerName="payload-processing"
Oct 30 13:37:20.778586 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.778514    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f79cd8-b784-4ef6-9a74-69a815c6e8a8" containerName="payload-pre-processing"
Oct 30 13:37:20.780981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.780960    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-podinfo\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.784543 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.784514    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"
Oct 30 13:37:20.792445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.792418    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5zj\" (UniqueName: \"kubernetes.io/projected/6e02b51d-4bfe-4805-b52e-d65419291f1e-kube-api-access-bt5zj\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.795720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.795696    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-token\") pod \"e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") " pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.799579 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.799558    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1c8130-dockercfg-dz2mk\""
Oct 30 13:37:20.808520 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.804765    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1c8130\""
Oct 30 13:37:20.808959 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.808915    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"]
Oct 30 13:37:20.842017 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.841993    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"]
Oct 30 13:37:20.848212 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.848184    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3c1b28-79b6b78954-bdnxz"]
Oct 30 13:37:20.867751 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.867726    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:20.873538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.873513    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8-kube-api-access-dczn5\" (UniqueName: \"kubernetes.io/projected/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8-kube-api-access-dczn5\") pod \"d3f79cd8-b784-4ef6-9a74-69a815c6e8a8\" (UID: \"d3f79cd8-b784-4ef6-9a74-69a815c6e8a8\") "
Oct 30 13:37:20.873762 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.873745    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01f1034a-cc31-43bc-91d4-8c0fc67e5960-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f1034a-cc31-43bc-91d4-8c0fc67e5960-plugins-config-volume\") pod \"01f1034a-cc31-43bc-91d4-8c0fc67e5960\" (UID: \"01f1034a-cc31-43bc-91d4-8c0fc67e5960\") "
Oct 30 13:37:20.874196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.873846    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01f1034a-cc31-43bc-91d4-8c0fc67e5960-kube-api-access-tckc8\" (UniqueName: \"kubernetes.io/projected/01f1034a-cc31-43bc-91d4-8c0fc67e5960-kube-api-access-tckc8\") pod \"01f1034a-cc31-43bc-91d4-8c0fc67e5960\" (UID: \"01f1034a-cc31-43bc-91d4-8c0fc67e5960\") "
Oct 30 13:37:20.874196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.873882    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8-plugins-config-volume\") pod \"d3f79cd8-b784-4ef6-9a74-69a815c6e8a8\" (UID: \"d3f79cd8-b784-4ef6-9a74-69a815c6e8a8\") "
Oct 30 13:37:20.874449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.874266    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01f1034a-cc31-43bc-91d4-8c0fc67e5960-plugins-config-volume" pod "01f1034a-cc31-43bc-91d4-8c0fc67e5960" (UID: "01f1034a-cc31-43bc-91d4-8c0fc67e5960"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:20.874449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.874351    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8-plugins-config-volume" pod "d3f79cd8-b784-4ef6-9a74-69a815c6e8a8" (UID: "d3f79cd8-b784-4ef6-9a74-69a815c6e8a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:20.875976 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.875950    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01f1034a-cc31-43bc-91d4-8c0fc67e5960-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:20.876114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.876000    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:20.876880 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.876844    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01f1034a-cc31-43bc-91d4-8c0fc67e5960-kube-api-access-tckc8" pod "01f1034a-cc31-43bc-91d4-8c0fc67e5960" (UID: "01f1034a-cc31-43bc-91d4-8c0fc67e5960"). InnerVolumeSpecName "kube-api-access-tckc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:20.877514 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.877494    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8-kube-api-access-dczn5" pod "d3f79cd8-b784-4ef6-9a74-69a815c6e8a8" (UID: "d3f79cd8-b784-4ef6-9a74-69a815c6e8a8"). InnerVolumeSpecName "kube-api-access-dczn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:20.997326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.981700    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58368e3f-ddb9-4363-9d05-93e5bb0e037e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn\" (UID: \"58368e3f-ddb9-4363-9d05-93e5bb0e037e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"
Oct 30 13:37:20.997326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.981930    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lbkd\" (UniqueName: \"kubernetes.io/projected/58368e3f-ddb9-4363-9d05-93e5bb0e037e-kube-api-access-2lbkd\") pod \"payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn\" (UID: \"58368e3f-ddb9-4363-9d05-93e5bb0e037e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"
Oct 30 13:37:20.997326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.982153    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tckc8\" (UniqueName: \"kubernetes.io/projected/01f1034a-cc31-43bc-91d4-8c0fc67e5960-kube-api-access-tckc8\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:20.997326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:20.982173    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dczn5\" (UniqueName: \"kubernetes.io/projected/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8-kube-api-access-dczn5\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:21.083968 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.083867    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lbkd\" (UniqueName: \"kubernetes.io/projected/58368e3f-ddb9-4363-9d05-93e5bb0e037e-kube-api-access-2lbkd\") pod \"payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn\" (UID: \"58368e3f-ddb9-4363-9d05-93e5bb0e037e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"
Oct 30 13:37:21.084225 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.084208    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58368e3f-ddb9-4363-9d05-93e5bb0e037e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn\" (UID: \"58368e3f-ddb9-4363-9d05-93e5bb0e037e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"
Oct 30 13:37:21.085129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.085106    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58368e3f-ddb9-4363-9d05-93e5bb0e037e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn\" (UID: \"58368e3f-ddb9-4363-9d05-93e5bb0e037e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"
Oct 30 13:37:21.106218 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.106191    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lbkd\" (UniqueName: \"kubernetes.io/projected/58368e3f-ddb9-4363-9d05-93e5bb0e037e-kube-api-access-2lbkd\") pod \"payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn\" (UID: \"58368e3f-ddb9-4363-9d05-93e5bb0e037e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"
Oct 30 13:37:21.115225 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.115193    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"
Oct 30 13:37:21.228003 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.227964    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"]
Oct 30 13:37:21.231904 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.231168    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:37:21.231904 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.231257    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:37:21.231904 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.231324    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:37:21.490652 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:37:21.490618    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58368e3f_ddb9_4363_9d05_93e5bb0e037e.slice/crio-165e7dad2873f5a9d19f90be8d5ae3aec7d20a8ac60075290cf96ee9c8d06db4 WatchSource:0}: Error finding container 165e7dad2873f5a9d19f90be8d5ae3aec7d20a8ac60075290cf96ee9c8d06db4: Status 404 returned error can't find the container with id 165e7dad2873f5a9d19f90be8d5ae3aec7d20a8ac60075290cf96ee9c8d06db4
Oct 30 13:37:21.490788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.490718    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"]
Oct 30 13:37:21.740160 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.740067    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"
Oct 30 13:37:21.740571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.740403    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n" event={"ID":"d3f79cd8-b784-4ef6-9a74-69a815c6e8a8","Type":"ContainerDied","Data":"ae1eae184c862dd85721c2c0f779a7a60bf7d76393cc480d5ba7a5fc322fc262"}
Oct 30 13:37:21.740571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.740474    5893 scope.go:122] "RemoveContainer" containerID="707286ec786736b8b5f55628f6379022f1ad10a0f654c8dd6c6462cbad4addce"
Oct 30 13:37:21.747521 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.744262    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn" event={"ID":"58368e3f-ddb9-4363-9d05-93e5bb0e037e","Type":"ContainerStarted","Data":"4e36f49b911129963f6258c1b6ed7ff13db843a911e7819679f2c3fd64daeacb"}
Oct 30 13:37:21.747521 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.747240    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"
Oct 30 13:37:21.747800 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.747265    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn" event={"ID":"58368e3f-ddb9-4363-9d05-93e5bb0e037e","Type":"ContainerStarted","Data":"165e7dad2873f5a9d19f90be8d5ae3aec7d20a8ac60075290cf96ee9c8d06db4"}
Oct 30 13:37:21.753291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.749650    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl" event={"ID":"01f1034a-cc31-43bc-91d4-8c0fc67e5960","Type":"ContainerDied","Data":"89a55d3fa262917bdc6fc0c69339827f304acfd1fb481f1cc5cca53737929c6f"}
Oct 30 13:37:21.753291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.749759    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"
Oct 30 13:37:21.782808 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.782760    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" event={"ID":"6e02b51d-4bfe-4805-b52e-d65419291f1e","Type":"ContainerStarted","Data":"810c3a6e5a613dee41706a5a570f87dc2b5e4c4a47b683e1575b29506ff32813"}
Oct 30 13:37:21.783006 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.782841    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" event={"ID":"6e02b51d-4bfe-4805-b52e-d65419291f1e","Type":"ContainerStarted","Data":"a51a3ed7cdf67de90cbfe278607fc547ba437163d6bf17c678157b5be3663f0c"}
Oct 30 13:37:21.793592 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.793568    5893 scope.go:122] "RemoveContainer" containerID="d16a7c588fa50f4f52ed027cda22c53cd17b743257f57419b7223e2853897648"
Oct 30 13:37:21.839255 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.839206    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn" podStartSLOduration=139.839188895 podStartE2EDuration="2m19.839188895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:21.835681558 +0000 UTC m=+3083.419111280" watchObservedRunningTime="2026-10-30 13:37:21.839188895 +0000 UTC m=+3083.422618589"
Oct 30 13:37:21.852350 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:37:21.852306    5893 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f1034a_cc31_43bc_91d4_8c0fc67e5960.slice/crio-89a55d3fa262917bdc6fc0c69339827f304acfd1fb481f1cc5cca53737929c6f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f79cd8_b784_4ef6_9a74_69a815c6e8a8.slice/crio-ae1eae184c862dd85721c2c0f779a7a60bf7d76393cc480d5ba7a5fc322fc262\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f1034a_cc31_43bc_91d4_8c0fc67e5960.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 13:37:21.868143 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.868109    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:21.884263 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.884226    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"]
Oct 30 13:37:21.891314 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.891283    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3c1b28-6546f8768-ggdfl"]
Oct 30 13:37:21.927776 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.927738    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"]
Oct 30 13:37:21.931744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.931719    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3c1b28-75964dbd7-hhr4n"]
Oct 30 13:37:21.979326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:21.979239    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" podStartSLOduration=114.979218198 podStartE2EDuration="1m54.979218198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:21.975737556 +0000 UTC m=+3083.559167302" watchObservedRunningTime="2026-10-30 13:37:21.979218198 +0000 UTC m=+3083.562647892"
Oct 30 13:37:22.535008 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:22.534975    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01f1034a-cc31-43bc-91d4-8c0fc67e5960" path="/var/lib/kubelet/pods/01f1034a-cc31-43bc-91d4-8c0fc67e5960/volumes"
Oct 30 13:37:22.535475 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:22.535461    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0bfdfdc-1f43-4127-b050-4e8a2c2f516f" path="/var/lib/kubelet/pods/c0bfdfdc-1f43-4127-b050-4e8a2c2f516f/volumes"
Oct 30 13:37:22.535874 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:22.535861    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f79cd8-b784-4ef6-9a74-69a815c6e8a8" path="/var/lib/kubelet/pods/d3f79cd8-b784-4ef6-9a74-69a815c6e8a8/volumes"
Oct 30 13:37:22.875125 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:22.875037    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:23.840330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:23.834490    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:23.852296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:23.846312    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:37:24.840916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:24.840886    5893 generic.go:356] "Generic (PLEG): container finished" podID="91c3dae9-4165-4660-bff5-f728025903cd" containerID="c1a79b3af28d0b470e7871b0474881f3c3512f5d3adba73c351966fd473f7504" exitCode=0
Oct 30 13:37:24.841388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:24.840959    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn" event={"ID":"91c3dae9-4165-4660-bff5-f728025903cd","Type":"ContainerDied","Data":"c1a79b3af28d0b470e7871b0474881f3c3512f5d3adba73c351966fd473f7504"}
Oct 30 13:37:24.889603 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:24.889172    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:37:24.966487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:24.966452    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c3dae9-4165-4660-bff5-f728025903cd-kube-api-access-qgttl\" (UniqueName: \"kubernetes.io/projected/91c3dae9-4165-4660-bff5-f728025903cd-kube-api-access-qgttl\") pod \"91c3dae9-4165-4660-bff5-f728025903cd\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") "
Oct 30 13:37:25.016476 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.016439    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c3dae9-4165-4660-bff5-f728025903cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c3dae9-4165-4660-bff5-f728025903cd-istiod-ca-cert\") pod \"91c3dae9-4165-4660-bff5-f728025903cd\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") "
Oct 30 13:37:25.016711 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.016692    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-credential-socket\") pod \"91c3dae9-4165-4660-bff5-f728025903cd\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") "
Oct 30 13:37:25.016919 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.016902    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-workload-certs\") pod \"91c3dae9-4165-4660-bff5-f728025903cd\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") "
Oct 30 13:37:25.017107 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.017091    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-workload-socket\") pod \"91c3dae9-4165-4660-bff5-f728025903cd\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") "
Oct 30 13:37:25.017358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.017341    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-istio-envoy\") pod \"91c3dae9-4165-4660-bff5-f728025903cd\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") "
Oct 30 13:37:25.017523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.017508    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c3dae9-4165-4660-bff5-f728025903cd-istio-token\" (UniqueName: \"kubernetes.io/projected/91c3dae9-4165-4660-bff5-f728025903cd-istio-token\") pod \"91c3dae9-4165-4660-bff5-f728025903cd\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") "
Oct 30 13:37:25.017739 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.017724    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-istio-data\") pod \"91c3dae9-4165-4660-bff5-f728025903cd\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") "
Oct 30 13:37:25.017870 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.017855    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91c3dae9-4165-4660-bff5-f728025903cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c3dae9-4165-4660-bff5-f728025903cd-istio-podinfo\") pod \"91c3dae9-4165-4660-bff5-f728025903cd\" (UID: \"91c3dae9-4165-4660-bff5-f728025903cd\") "
Oct 30 13:37:25.027796 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.026954    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-workload-certs" pod "91c3dae9-4165-4660-bff5-f728025903cd" (UID: "91c3dae9-4165-4660-bff5-f728025903cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:25.027796 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.027159    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-credential-socket" pod "91c3dae9-4165-4660-bff5-f728025903cd" (UID: "91c3dae9-4165-4660-bff5-f728025903cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:25.027796 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.027352    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-workload-socket" pod "91c3dae9-4165-4660-bff5-f728025903cd" (UID: "91c3dae9-4165-4660-bff5-f728025903cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:25.044095 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.035451    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c3dae9-4165-4660-bff5-f728025903cd-istiod-ca-cert" pod "91c3dae9-4165-4660-bff5-f728025903cd" (UID: "91c3dae9-4165-4660-bff5-f728025903cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:25.044095 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.042352    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-istio-data" pod "91c3dae9-4165-4660-bff5-f728025903cd" (UID: "91c3dae9-4165-4660-bff5-f728025903cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:25.058509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.055962    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c3dae9-4165-4660-bff5-f728025903cd-kube-api-access-qgttl" pod "91c3dae9-4165-4660-bff5-f728025903cd" (UID: "91c3dae9-4165-4660-bff5-f728025903cd"). InnerVolumeSpecName "kube-api-access-qgttl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:25.081457 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.081418    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"]
Oct 30 13:37:25.082810 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.082778    5893 cpu_manager.go:414] "containerMap: removing container" podUID="91c3dae9-4165-4660-bff5-f728025903cd" containerName="istio-proxy"
Oct 30 13:37:25.082810 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.082810    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c3dae9-4165-4660-bff5-f728025903cd" containerName="istio-proxy"
Oct 30 13:37:25.083118 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.083095    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c3dae9-4165-4660-bff5-f728025903cd" containerName="istio-proxy"
Oct 30 13:37:25.102191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.098567    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"
Oct 30 13:37:25.118518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.116168    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"]
Oct 30 13:37:25.136547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.130821    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c3dae9-4165-4660-bff5-f728025903cd-istio-token" pod "91c3dae9-4165-4660-bff5-f728025903cd" (UID: "91c3dae9-4165-4660-bff5-f728025903cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:25.158336 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.157006    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-istio-envoy" pod "91c3dae9-4165-4660-bff5-f728025903cd" (UID: "91c3dae9-4165-4660-bff5-f728025903cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:25.159861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.159806    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91c3dae9-4165-4660-bff5-f728025903cd-istio-podinfo" pod "91c3dae9-4165-4660-bff5-f728025903cd" (UID: "91c3dae9-4165-4660-bff5-f728025903cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:37:25.160129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.160071    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c3dae9-4165-4660-bff5-f728025903cd-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:25.160129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.160092    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:25.160129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.160107    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:25.160129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.160120    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:25.160440 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.160133    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:25.160440 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.160148    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c3dae9-4165-4660-bff5-f728025903cd-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:25.160440 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.160159    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c3dae9-4165-4660-bff5-f728025903cd-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:25.160440 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.160171    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c3dae9-4165-4660-bff5-f728025903cd-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:25.160440 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.160185    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgttl\" (UniqueName: \"kubernetes.io/projected/91c3dae9-4165-4660-bff5-f728025903cd-kube-api-access-qgttl\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:25.261951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.261918    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd8433a-82e7-43f7-a036-1c79a64230a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js\" (UID: \"ddd8433a-82e7-43f7-a036-1c79a64230a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"
Oct 30 13:37:25.262145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.262082    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8lkv\" (UniqueName: \"kubernetes.io/projected/ddd8433a-82e7-43f7-a036-1c79a64230a1-kube-api-access-q8lkv\") pod \"payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js\" (UID: \"ddd8433a-82e7-43f7-a036-1c79a64230a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"
Oct 30 13:37:25.364582 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.364491    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8lkv\" (UniqueName: \"kubernetes.io/projected/ddd8433a-82e7-43f7-a036-1c79a64230a1-kube-api-access-q8lkv\") pod \"payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js\" (UID: \"ddd8433a-82e7-43f7-a036-1c79a64230a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"
Oct 30 13:37:25.364873 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.364857    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd8433a-82e7-43f7-a036-1c79a64230a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js\" (UID: \"ddd8433a-82e7-43f7-a036-1c79a64230a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"
Oct 30 13:37:25.365577 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.365557    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd8433a-82e7-43f7-a036-1c79a64230a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js\" (UID: \"ddd8433a-82e7-43f7-a036-1c79a64230a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"
Oct 30 13:37:25.385773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.385742    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8lkv\" (UniqueName: \"kubernetes.io/projected/ddd8433a-82e7-43f7-a036-1c79a64230a1-kube-api-access-q8lkv\") pod \"payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js\" (UID: \"ddd8433a-82e7-43f7-a036-1c79a64230a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"
Oct 30 13:37:25.484607 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.484572    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"
Oct 30 13:37:25.793813 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.793783    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"]
Oct 30 13:37:25.796701 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:37:25.796670    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd8433a_82e7_43f7_a036_1c79a64230a1.slice/crio-fb56a5d7cf7f476a47a2891fd7f64c9e52ed2b29fca9aa9cb632a6a85028c19a WatchSource:0}: Error finding container fb56a5d7cf7f476a47a2891fd7f64c9e52ed2b29fca9aa9cb632a6a85028c19a: Status 404 returned error can't find the container with id fb56a5d7cf7f476a47a2891fd7f64c9e52ed2b29fca9aa9cb632a6a85028c19a
Oct 30 13:37:25.851562 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.851534    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"
Oct 30 13:37:25.852186 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.851568    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn" event={"ID":"91c3dae9-4165-4660-bff5-f728025903cd","Type":"ContainerDied","Data":"13fdfc9cda652e0c98d87c741d10dd1dc192ca4a5e0a9597a718b7be7cac2023"}
Oct 30 13:37:25.852186 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.851634    5893 scope.go:122] "RemoveContainer" containerID="c1a79b3af28d0b470e7871b0474881f3c3512f5d3adba73c351966fd473f7504"
Oct 30 13:37:25.853744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.853659    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js" event={"ID":"ddd8433a-82e7-43f7-a036-1c79a64230a1","Type":"ContainerStarted","Data":"fb56a5d7cf7f476a47a2891fd7f64c9e52ed2b29fca9aa9cb632a6a85028c19a"}
Oct 30 13:37:25.901239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.901216    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"]
Oct 30 13:37:25.912195 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:25.912157    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-96ab73-openshift-default-586f58856c-cvlmn"]
Oct 30 13:37:26.536043 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.536013    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c3dae9-4165-4660-bff5-f728025903cd" path="/var/lib/kubelet/pods/91c3dae9-4165-4660-bff5-f728025903cd/volumes"
Oct 30 13:37:26.559120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.559089    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"]
Oct 30 13:37:26.564337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.564312    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"
Oct 30 13:37:26.584335 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.584312    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ba8145-dockercfg-nqqh7\""
Oct 30 13:37:26.586120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.586091    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fppd\" (UniqueName: \"kubernetes.io/projected/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f-kube-api-access-5fppd\") pod \"payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z\" (UID: \"2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"
Oct 30 13:37:26.586487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.586459    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z\" (UID: \"2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"
Oct 30 13:37:26.590705 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.590684    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"]
Oct 30 13:37:26.617574 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.617538    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"]
Oct 30 13:37:26.627435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.626339    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"
Oct 30 13:37:26.636084 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.636056    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ba8145\""
Oct 30 13:37:26.651581 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.651544    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"]
Oct 30 13:37:26.706180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.706025    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z\" (UID: \"2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"
Oct 30 13:37:26.706587 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.706557    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09976b18-da28-4c48-8ac1-c3f914e632b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j\" (UID: \"09976b18-da28-4c48-8ac1-c3f914e632b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"
Oct 30 13:37:26.706998 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.706978    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbmlp\" (UniqueName: \"kubernetes.io/projected/09976b18-da28-4c48-8ac1-c3f914e632b4-kube-api-access-cbmlp\") pod \"payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j\" (UID: \"09976b18-da28-4c48-8ac1-c3f914e632b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"
Oct 30 13:37:26.707191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.707173    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fppd\" (UniqueName: \"kubernetes.io/projected/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f-kube-api-access-5fppd\") pod \"payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z\" (UID: \"2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"
Oct 30 13:37:26.708915 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.708887    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z\" (UID: \"2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"
Oct 30 13:37:26.836351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.811128    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09976b18-da28-4c48-8ac1-c3f914e632b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j\" (UID: \"09976b18-da28-4c48-8ac1-c3f914e632b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"
Oct 30 13:37:26.847296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.839814    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbmlp\" (UniqueName: \"kubernetes.io/projected/09976b18-da28-4c48-8ac1-c3f914e632b4-kube-api-access-cbmlp\") pod \"payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j\" (UID: \"09976b18-da28-4c48-8ac1-c3f914e632b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"
Oct 30 13:37:26.872299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.839440    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fppd\" (UniqueName: \"kubernetes.io/projected/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f-kube-api-access-5fppd\") pod \"payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z\" (UID: \"2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"
Oct 30 13:37:26.874436 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.838264    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09976b18-da28-4c48-8ac1-c3f914e632b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j\" (UID: \"09976b18-da28-4c48-8ac1-c3f914e632b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"
Oct 30 13:37:26.897299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.880692    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"
Oct 30 13:37:26.919701 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.918497    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbmlp\" (UniqueName: \"kubernetes.io/projected/09976b18-da28-4c48-8ac1-c3f914e632b4-kube-api-access-cbmlp\") pod \"payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j\" (UID: \"09976b18-da28-4c48-8ac1-c3f914e632b4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"
Oct 30 13:37:26.947300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.943406    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js" event={"ID":"ddd8433a-82e7-43f7-a036-1c79a64230a1","Type":"ContainerStarted","Data":"2ddd29bf0e636f56dd14a32ed6edce1aba855c0f977142dea71556fd134626f6"}
Oct 30 13:37:26.947300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.947266    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"
Oct 30 13:37:26.957895 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:26.955156    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"
Oct 30 13:37:27.004208 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:27.004137    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js" podStartSLOduration=172.004116598 podStartE2EDuration="2m52.004116598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:26.988691826 +0000 UTC m=+3088.572121519" watchObservedRunningTime="2026-10-30 13:37:27.004116598 +0000 UTC m=+3088.587546292"
Oct 30 13:37:27.310389 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:27.310356    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"]
Oct 30 13:37:27.312894 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:37:27.312858    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09976b18_da28_4c48_8ac1_c3f914e632b4.slice/crio-3b693caeb0bc02f47a1aed8307312a70f29ca1de3bc459a959860d2dd49128c7 WatchSource:0}: Error finding container 3b693caeb0bc02f47a1aed8307312a70f29ca1de3bc459a959860d2dd49128c7: Status 404 returned error can't find the container with id 3b693caeb0bc02f47a1aed8307312a70f29ca1de3bc459a959860d2dd49128c7
Oct 30 13:37:27.328343 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:27.328322    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"]
Oct 30 13:37:27.331995 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:37:27.331970    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4edd3c_438c_4c1e_90ea_3ef81b2ad02f.slice/crio-e3b31544de5cc5d4cd64227908a0f04dab5eb5f2f89aae1a090926bbd6437652 WatchSource:0}: Error finding container e3b31544de5cc5d4cd64227908a0f04dab5eb5f2f89aae1a090926bbd6437652: Status 404 returned error can't find the container with id e3b31544de5cc5d4cd64227908a0f04dab5eb5f2f89aae1a090926bbd6437652
Oct 30 13:37:27.962333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:27.962250    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z" event={"ID":"2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f","Type":"ContainerStarted","Data":"b1bb6868e44657b01709e7e68add5abc6f877bd3e37ace3ec13af324d849248e"}
Oct 30 13:37:27.962812 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:27.962366    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z" event={"ID":"2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f","Type":"ContainerStarted","Data":"e3b31544de5cc5d4cd64227908a0f04dab5eb5f2f89aae1a090926bbd6437652"}
Oct 30 13:37:27.962812 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:27.962402    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"
Oct 30 13:37:27.964547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:27.964516    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j" event={"ID":"09976b18-da28-4c48-8ac1-c3f914e632b4","Type":"ContainerStarted","Data":"6c8a13ecaaa729cc1852e867e97f9951a01e8fc7e322cd278f454424c3ff1af7"}
Oct 30 13:37:27.964717 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:27.964565    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j" event={"ID":"09976b18-da28-4c48-8ac1-c3f914e632b4","Type":"ContainerStarted","Data":"3b693caeb0bc02f47a1aed8307312a70f29ca1de3bc459a959860d2dd49128c7"}
Oct 30 13:37:27.964717 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:27.964606    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"
Oct 30 13:37:28.001765 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:28.001716    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z" podStartSLOduration=2.00170162 podStartE2EDuration="2.00170162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:27.999622883 +0000 UTC m=+3089.583052576" watchObservedRunningTime="2026-10-30 13:37:28.00170162 +0000 UTC m=+3089.585131312"
Oct 30 13:37:28.055454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:28.055387    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j" podStartSLOduration=2.055365968 podStartE2EDuration="2.055365968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:28.047862034 +0000 UTC m=+3089.631291730" watchObservedRunningTime="2026-10-30 13:37:28.055365968 +0000 UTC m=+3089.638795661"
Oct 30 13:37:28.696680 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:28.696641    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"]
Oct 30 13:37:28.697055 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:28.696955    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs" podUID="5e392435-687c-4ead-b5c2-44d275dd61d5" containerName="istio-proxy" containerID="cri-o://2891664e4348d22eeac18f21e22de7412af541e7c486743763bb12331509628c" gracePeriod=30
Oct 30 13:37:28.706041 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:28.706011    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"]
Oct 30 13:37:28.706309 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:28.706265    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy" containerID="cri-o://cc89889bf848e635745624ed2b9657ef2f6b3f3cdc9263eb66e96099548f90d7" gracePeriod=30
Oct 30 13:37:28.723825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:28.723780    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"]
Oct 30 13:37:28.724089 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:28.724064    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj" podUID="30fc4668-2aed-4383-b5b1-092a5f3411a5" containerName="istio-proxy" containerID="cri-o://b39228adeb9377ff748803c785eff1b614622a8d92da47526c198440ead4224c" gracePeriod=30
Oct 30 13:37:31.407210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:31.407172    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:37:31.407633 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:31.407226    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:37:32.803905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:32.803873    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"
Oct 30 13:37:34.066726 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.066680    5893 generic.go:356] "Generic (PLEG): container finished" podID="5e392435-687c-4ead-b5c2-44d275dd61d5" containerID="2891664e4348d22eeac18f21e22de7412af541e7c486743763bb12331509628c" exitCode=0
Oct 30 13:37:34.067445 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.067415    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs" event={"ID":"5e392435-687c-4ead-b5c2-44d275dd61d5","Type":"ContainerDied","Data":"2891664e4348d22eeac18f21e22de7412af541e7c486743763bb12331509628c"}
Oct 30 13:37:34.117523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.111386    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj" event={"ID":"30fc4668-2aed-4383-b5b1-092a5f3411a5","Type":"ContainerDied","Data":"b39228adeb9377ff748803c785eff1b614622a8d92da47526c198440ead4224c"}
Oct 30 13:37:34.117523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.111343    5893 generic.go:356] "Generic (PLEG): container finished" podID="30fc4668-2aed-4383-b5b1-092a5f3411a5" containerID="b39228adeb9377ff748803c785eff1b614622a8d92da47526c198440ead4224c" exitCode=0
Oct 30 13:37:34.117523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.116778    5893 generic.go:356] "Generic (PLEG): container finished" podID="262ca860-a303-4568-935a-bb158b4dc3f3" containerID="cc89889bf848e635745624ed2b9657ef2f6b3f3cdc9263eb66e96099548f90d7" exitCode=0
Oct 30 13:37:34.117523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.116922    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" event={"ID":"262ca860-a303-4568-935a-bb158b4dc3f3","Type":"ContainerDied","Data":"cc89889bf848e635745624ed2b9657ef2f6b3f3cdc9263eb66e96099548f90d7"}
Oct 30 13:37:34.117523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.116996    5893 scope.go:122] "RemoveContainer" containerID="4cb1a2da5908989966a92e685ca67dcd5fac3da34a3a72e94d18d65796d786f8"
Oct 30 13:37:34.159145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.159112    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:37:34.187833 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.187804    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:37:34.189408 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.189385    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:37:34.207841 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.207242    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-credential-socket\") pod \"30fc4668-2aed-4383-b5b1-092a5f3411a5\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") "
Oct 30 13:37:34.207841 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.207541    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-credential-socket" pod "30fc4668-2aed-4383-b5b1-092a5f3411a5" (UID: "30fc4668-2aed-4383-b5b1-092a5f3411a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.207841 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.207716    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-token\" (UniqueName: \"kubernetes.io/projected/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-token\") pod \"30fc4668-2aed-4383-b5b1-092a5f3411a5\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") "
Oct 30 13:37:34.207841 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.207744    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30fc4668-2aed-4383-b5b1-092a5f3411a5-kube-api-access-vsm8t\" (UniqueName: \"kubernetes.io/projected/30fc4668-2aed-4383-b5b1-092a5f3411a5-kube-api-access-vsm8t\") pod \"30fc4668-2aed-4383-b5b1-092a5f3411a5\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") "
Oct 30 13:37:34.208171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.207886    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-workload-socket\") pod \"30fc4668-2aed-4383-b5b1-092a5f3411a5\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") "
Oct 30 13:37:34.208171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.207990    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-podinfo\") pod \"30fc4668-2aed-4383-b5b1-092a5f3411a5\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") "
Oct 30 13:37:34.208171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.208037    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-data\") pod \"30fc4668-2aed-4383-b5b1-092a5f3411a5\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") "
Oct 30 13:37:34.208171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.208115    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-workload-certs\") pod \"30fc4668-2aed-4383-b5b1-092a5f3411a5\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") "
Oct 30 13:37:34.208171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.208154    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-envoy\") pod \"30fc4668-2aed-4383-b5b1-092a5f3411a5\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") "
Oct 30 13:37:34.209837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.208259    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30fc4668-2aed-4383-b5b1-092a5f3411a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30fc4668-2aed-4383-b5b1-092a5f3411a5-istiod-ca-cert\") pod \"30fc4668-2aed-4383-b5b1-092a5f3411a5\" (UID: \"30fc4668-2aed-4383-b5b1-092a5f3411a5\") "
Oct 30 13:37:34.209837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.209298    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.209837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.209616    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30fc4668-2aed-4383-b5b1-092a5f3411a5-istiod-ca-cert" pod "30fc4668-2aed-4383-b5b1-092a5f3411a5" (UID: "30fc4668-2aed-4383-b5b1-092a5f3411a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:34.210020 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.209884    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-data" pod "30fc4668-2aed-4383-b5b1-092a5f3411a5" (UID: "30fc4668-2aed-4383-b5b1-092a5f3411a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.216178 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.210376    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-workload-certs" pod "30fc4668-2aed-4383-b5b1-092a5f3411a5" (UID: "30fc4668-2aed-4383-b5b1-092a5f3411a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.225863 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.223038    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-workload-socket" pod "30fc4668-2aed-4383-b5b1-092a5f3411a5" (UID: "30fc4668-2aed-4383-b5b1-092a5f3411a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.237833 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.237227    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-podinfo" pod "30fc4668-2aed-4383-b5b1-092a5f3411a5" (UID: "30fc4668-2aed-4383-b5b1-092a5f3411a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:37:34.241068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.241014    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-token" pod "30fc4668-2aed-4383-b5b1-092a5f3411a5" (UID: "30fc4668-2aed-4383-b5b1-092a5f3411a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:34.241226 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.241158    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30fc4668-2aed-4383-b5b1-092a5f3411a5-kube-api-access-vsm8t" pod "30fc4668-2aed-4383-b5b1-092a5f3411a5" (UID: "30fc4668-2aed-4383-b5b1-092a5f3411a5"). InnerVolumeSpecName "kube-api-access-vsm8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:34.242638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.242603    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-envoy" pod "30fc4668-2aed-4383-b5b1-092a5f3411a5" (UID: "30fc4668-2aed-4383-b5b1-092a5f3411a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.245207 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.245180    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"]
Oct 30 13:37:34.246773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.246700    5893 cpu_manager.go:414] "containerMap: removing container" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy"
Oct 30 13:37:34.246773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.246728    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy"
Oct 30 13:37:34.247949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.246783    5893 cpu_manager.go:414] "containerMap: removing container" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy"
Oct 30 13:37:34.247949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.246790    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy"
Oct 30 13:37:34.247949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.246803    5893 cpu_manager.go:414] "containerMap: removing container" podUID="30fc4668-2aed-4383-b5b1-092a5f3411a5" containerName="istio-proxy"
Oct 30 13:37:34.247949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.246810    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30fc4668-2aed-4383-b5b1-092a5f3411a5" containerName="istio-proxy"
Oct 30 13:37:34.247949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.246843    5893 cpu_manager.go:414] "containerMap: removing container" podUID="5e392435-687c-4ead-b5c2-44d275dd61d5" containerName="istio-proxy"
Oct 30 13:37:34.247949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.246850    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e392435-687c-4ead-b5c2-44d275dd61d5" containerName="istio-proxy"
Oct 30 13:37:34.247949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.247064    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="30fc4668-2aed-4383-b5b1-092a5f3411a5" containerName="istio-proxy"
Oct 30 13:37:34.247949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.247084    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e392435-687c-4ead-b5c2-44d275dd61d5" containerName="istio-proxy"
Oct 30 13:37:34.247949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.247110    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy"
Oct 30 13:37:34.267356 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.263906    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:34.271652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.271622    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"]
Oct 30 13:37:34.285903 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.285874    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"]
Oct 30 13:37:34.287825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.287805    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" containerName="istio-proxy"
Oct 30 13:37:34.293187 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.293167    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.311019 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.310979    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/262ca860-a303-4568-935a-bb158b4dc3f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262ca860-a303-4568-935a-bb158b4dc3f3-istio-podinfo\") pod \"262ca860-a303-4568-935a-bb158b4dc3f3\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") "
Oct 30 13:37:34.313069 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.312371    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e392435-687c-4ead-b5c2-44d275dd61d5-kube-api-access-xjpx5\" (UniqueName: \"kubernetes.io/projected/5e392435-687c-4ead-b5c2-44d275dd61d5-kube-api-access-xjpx5\") pod \"5e392435-687c-4ead-b5c2-44d275dd61d5\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") "
Oct 30 13:37:34.314239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.313597    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-credential-socket\") pod \"5e392435-687c-4ead-b5c2-44d275dd61d5\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") "
Oct 30 13:37:34.315244 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.314618    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262ca860-a303-4568-935a-bb158b4dc3f3-istio-token\" (UniqueName: \"kubernetes.io/projected/262ca860-a303-4568-935a-bb158b4dc3f3-istio-token\") pod \"262ca860-a303-4568-935a-bb158b4dc3f3\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") "
Oct 30 13:37:34.315490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.315471    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-data\") pod \"5e392435-687c-4ead-b5c2-44d275dd61d5\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") "
Oct 30 13:37:34.315624 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.315608    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-podinfo\") pod \"5e392435-687c-4ead-b5c2-44d275dd61d5\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") "
Oct 30 13:37:34.315763 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.315745    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-workload-socket\") pod \"262ca860-a303-4568-935a-bb158b4dc3f3\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") "
Oct 30 13:37:34.315884 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.315870    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-workload-certs\") pod \"262ca860-a303-4568-935a-bb158b4dc3f3\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") "
Oct 30 13:37:34.316012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.316001    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262ca860-a303-4568-935a-bb158b4dc3f3-kube-api-access-6hdxl\" (UniqueName: \"kubernetes.io/projected/262ca860-a303-4568-935a-bb158b4dc3f3-kube-api-access-6hdxl\") pod \"262ca860-a303-4568-935a-bb158b4dc3f3\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") "
Oct 30 13:37:34.316603 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.316153    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-token\" (UniqueName: \"kubernetes.io/projected/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-token\") pod \"5e392435-687c-4ead-b5c2-44d275dd61d5\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") "
Oct 30 13:37:34.316833 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.316819    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-istio-envoy\") pod \"262ca860-a303-4568-935a-bb158b4dc3f3\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") "
Oct 30 13:37:34.316933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.316923    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262ca860-a303-4568-935a-bb158b4dc3f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262ca860-a303-4568-935a-bb158b4dc3f3-istiod-ca-cert\") pod \"262ca860-a303-4568-935a-bb158b4dc3f3\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") "
Oct 30 13:37:34.317011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.317002    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-istio-data\") pod \"262ca860-a303-4568-935a-bb158b4dc3f3\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") "
Oct 30 13:37:34.317127 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.317113    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-workload-certs\") pod \"5e392435-687c-4ead-b5c2-44d275dd61d5\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") "
Oct 30 13:37:34.323664 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.323625    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-envoy\") pod \"5e392435-687c-4ead-b5c2-44d275dd61d5\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") "
Oct 30 13:37:34.323793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.323686    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-credential-socket\") pod \"262ca860-a303-4568-935a-bb158b4dc3f3\" (UID: \"262ca860-a303-4568-935a-bb158b4dc3f3\") "
Oct 30 13:37:34.323864 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.323799    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e392435-687c-4ead-b5c2-44d275dd61d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e392435-687c-4ead-b5c2-44d275dd61d5-istiod-ca-cert\") pod \"5e392435-687c-4ead-b5c2-44d275dd61d5\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") "
Oct 30 13:37:34.323927 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.323885    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-workload-socket\") pod \"5e392435-687c-4ead-b5c2-44d275dd61d5\" (UID: \"5e392435-687c-4ead-b5c2-44d275dd61d5\") "
Oct 30 13:37:34.324329 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.324266    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ac8d101-9ee3-42fa-9719-66220e201923-metrics-certs\") pod \"maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:34.324429 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.324390    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ac8d101-9ee3-42fa-9719-66220e201923-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:34.324691 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.324633    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pshmc\" (UniqueName: \"kubernetes.io/projected/8ac8d101-9ee3-42fa-9719-66220e201923-kube-api-access-pshmc\") pod \"maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:34.324813 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.315503    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-credential-socket" pod "5e392435-687c-4ead-b5c2-44d275dd61d5" (UID: "5e392435-687c-4ead-b5c2-44d275dd61d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.324889 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.324870    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac8d101-9ee3-42fa-9719-66220e201923-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:34.324941 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.317520    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-workload-certs" pod "5e392435-687c-4ead-b5c2-44d275dd61d5" (UID: "5e392435-687c-4ead-b5c2-44d275dd61d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.324984 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.324795    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-data" pod "5e392435-687c-4ead-b5c2-44d275dd61d5" (UID: "5e392435-687c-4ead-b5c2-44d275dd61d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.325240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.325132    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30fc4668-2aed-4383-b5b1-092a5f3411a5-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.325240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.325156    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.325240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.325171    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.325240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.325186    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsm8t\" (UniqueName: \"kubernetes.io/projected/30fc4668-2aed-4383-b5b1-092a5f3411a5-kube-api-access-vsm8t\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.325240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.325201    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.325240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.325215    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.325240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.325227    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.325542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.325242    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.325542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.325257    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.325542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.325288    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30fc4668-2aed-4383-b5b1-092a5f3411a5-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.328304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.328259    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-credential-socket" pod "262ca860-a303-4568-935a-bb158b4dc3f3" (UID: "262ca860-a303-4568-935a-bb158b4dc3f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.328724 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.328701    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e392435-687c-4ead-b5c2-44d275dd61d5-istiod-ca-cert" pod "5e392435-687c-4ead-b5c2-44d275dd61d5" (UID: "5e392435-687c-4ead-b5c2-44d275dd61d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:34.329038 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.328999    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-istio-data" pod "262ca860-a303-4568-935a-bb158b4dc3f3" (UID: "262ca860-a303-4568-935a-bb158b4dc3f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.329259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.329237    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-workload-socket" pod "5e392435-687c-4ead-b5c2-44d275dd61d5" (UID: "5e392435-687c-4ead-b5c2-44d275dd61d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.331339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.329405    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-workload-certs" pod "262ca860-a303-4568-935a-bb158b4dc3f3" (UID: "262ca860-a303-4568-935a-bb158b4dc3f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.332702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.329417    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262ca860-a303-4568-935a-bb158b4dc3f3-istiod-ca-cert" pod "262ca860-a303-4568-935a-bb158b4dc3f3" (UID: "262ca860-a303-4568-935a-bb158b4dc3f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:34.332702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.329629    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-workload-socket" pod "262ca860-a303-4568-935a-bb158b4dc3f3" (UID: "262ca860-a303-4568-935a-bb158b4dc3f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.332702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.317044    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3b47e7-serving-cert\""
Oct 30 13:37:34.335319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.335193    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"]
Oct 30 13:37:34.336090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.336040    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-podinfo" pod "5e392435-687c-4ead-b5c2-44d275dd61d5" (UID: "5e392435-687c-4ead-b5c2-44d275dd61d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:37:34.338944 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.338912    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262ca860-a303-4568-935a-bb158b4dc3f3-kube-api-access-6hdxl" pod "262ca860-a303-4568-935a-bb158b4dc3f3" (UID: "262ca860-a303-4568-935a-bb158b4dc3f3"). InnerVolumeSpecName "kube-api-access-6hdxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:34.343745 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.343718    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-istio-envoy" pod "262ca860-a303-4568-935a-bb158b4dc3f3" (UID: "262ca860-a303-4568-935a-bb158b4dc3f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.344006 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.343927    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-token" pod "5e392435-687c-4ead-b5c2-44d275dd61d5" (UID: "5e392435-687c-4ead-b5c2-44d275dd61d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:34.344006 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.343981    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262ca860-a303-4568-935a-bb158b4dc3f3-istio-token" pod "262ca860-a303-4568-935a-bb158b4dc3f3" (UID: "262ca860-a303-4568-935a-bb158b4dc3f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:34.344427 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.344404    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e392435-687c-4ead-b5c2-44d275dd61d5-kube-api-access-xjpx5" pod "5e392435-687c-4ead-b5c2-44d275dd61d5" (UID: "5e392435-687c-4ead-b5c2-44d275dd61d5"). InnerVolumeSpecName "kube-api-access-xjpx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:34.345650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.345627    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/262ca860-a303-4568-935a-bb158b4dc3f3-istio-podinfo" pod "262ca860-a303-4568-935a-bb158b4dc3f3" (UID: "262ca860-a303-4568-935a-bb158b4dc3f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:37:34.346160 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.346133    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-envoy" pod "5e392435-687c-4ead-b5c2-44d275dd61d5" (UID: "5e392435-687c-4ead-b5c2-44d275dd61d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:34.346668 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.346653    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"]
Oct 30 13:37:34.362483 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.362439    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"]
Oct 30 13:37:34.369303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.362744    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.432826 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.432788    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pshmc\" (UniqueName: \"kubernetes.io/projected/8ac8d101-9ee3-42fa-9719-66220e201923-kube-api-access-pshmc\") pod \"maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:34.433023 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.432854    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.433023 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.432923    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.433023 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.432968    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433040    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac8d101-9ee3-42fa-9719-66220e201923-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433062    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728d41af-901f-4def-a2c1-aee79df80e1e-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433091    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433186    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433243    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433289    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728d41af-901f-4def-a2c1-aee79df80e1e-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433316    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433343    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433369    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433401    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728d41af-901f-4def-a2c1-aee79df80e1e-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433433    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433467    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ac8d101-9ee3-42fa-9719-66220e201923-metrics-certs\") pod \"maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433535    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433566    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ac8d101-9ee3-42fa-9719-66220e201923-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433608    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn2f6\" (UniqueName: \"kubernetes.io/projected/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-kube-api-access-bn2f6\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433636    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433661    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54tms\" (UniqueName: \"kubernetes.io/projected/728d41af-901f-4def-a2c1-aee79df80e1e-kube-api-access-54tms\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433754    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433872    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262ca860-a303-4568-935a-bb158b4dc3f3-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433888    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433902    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433917    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433932    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433948    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hdxl\" (UniqueName: \"kubernetes.io/projected/262ca860-a303-4568-935a-bb158b4dc3f3-kube-api-access-6hdxl\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433962    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433974    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433986    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262ca860-a303-4568-935a-bb158b4dc3f3-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.433998    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.434011    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.434023    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262ca860-a303-4568-935a-bb158b4dc3f3-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.434036    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e392435-687c-4ead-b5c2-44d275dd61d5-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.434048    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e392435-687c-4ead-b5c2-44d275dd61d5-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.434060    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262ca860-a303-4568-935a-bb158b4dc3f3-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.434493 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.434072    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjpx5\" (UniqueName: \"kubernetes.io/projected/5e392435-687c-4ead-b5c2-44d275dd61d5-kube-api-access-xjpx5\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:34.439134 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.435150    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac8d101-9ee3-42fa-9719-66220e201923-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:34.439134 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.439012    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ac8d101-9ee3-42fa-9719-66220e201923-metrics-certs\") pod \"maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:34.439134 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.439124    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ac8d101-9ee3-42fa-9719-66220e201923-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:34.465329 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.465301    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pshmc\" (UniqueName: \"kubernetes.io/projected/8ac8d101-9ee3-42fa-9719-66220e201923-kube-api-access-pshmc\") pod \"maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:34.535593 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.535511    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.535786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.535597    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.535786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.535640    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.535786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.535669    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.535786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.535716    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728d41af-901f-4def-a2c1-aee79df80e1e-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.535786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.535737    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.536718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.535805    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.536718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.535840    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.536718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.535857    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728d41af-901f-4def-a2c1-aee79df80e1e-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.536718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.535879    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.536718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.535898    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.536718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.535915    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.536718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.535936    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728d41af-901f-4def-a2c1-aee79df80e1e-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.536718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.535955    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.536718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.536001    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.536718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.536031    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn2f6\" (UniqueName: \"kubernetes.io/projected/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-kube-api-access-bn2f6\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.536718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.536049    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.536718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.536063    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54tms\" (UniqueName: \"kubernetes.io/projected/728d41af-901f-4def-a2c1-aee79df80e1e-kube-api-access-54tms\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.536718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.536632    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.537695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.537664    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.538478 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.538450    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728d41af-901f-4def-a2c1-aee79df80e1e-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.539454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.539426    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.539688 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.539669    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728d41af-901f-4def-a2c1-aee79df80e1e-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.539970 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.539946    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.540247 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.540204    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-credential-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.540461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.540437    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-workload-certs\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.540894 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.540867    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istiod-ca-cert\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.541067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.541048    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.541171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.541146    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-data\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.541298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.541248    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-podinfo\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.541557 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.541537    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-workload-socket\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.549952 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.549932    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-istio-envoy\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.558535 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.558497    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.561157 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.561135    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54tms\" (UniqueName: \"kubernetes.io/projected/728d41af-901f-4def-a2c1-aee79df80e1e-kube-api-access-54tms\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.562012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.561984    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728d41af-901f-4def-a2c1-aee79df80e1e-istio-token\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.562705 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.562683    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn2f6\" (UniqueName: \"kubernetes.io/projected/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-kube-api-access-bn2f6\") pod \"e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") " pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.602859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.602827    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:34.638379 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.638341    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:34.694727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.691851    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:34.936307 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.935719    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"]
Oct 30 13:37:34.939594 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:37:34.939551    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ac8d101_9ee3_42fa_9719_66220e201923.slice/crio-ca4e2499134c7651dea0b9b3e4b1a5a77ca453d892d2348d4dd34107bf434fd0 WatchSource:0}: Error finding container ca4e2499134c7651dea0b9b3e4b1a5a77ca453d892d2348d4dd34107bf434fd0: Status 404 returned error can't find the container with id ca4e2499134c7651dea0b9b3e4b1a5a77ca453d892d2348d4dd34107bf434fd0
Oct 30 13:37:34.942903 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.941642    5893 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:37:34.999103 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.999040    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:37:34.999312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:34.999132    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj" podUID="30fc4668-2aed-4383-b5b1-092a5f3411a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:37:35.082741 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.082717    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"]
Oct 30 13:37:35.086021 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:37:35.085990    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7e2c2e2_f1c3_4e05_8366_135e54ad4eaa.slice/crio-dc982469c0af36a148cf29aebdcdab78aaff8e4ca4272bc922d4ce1565ca3971 WatchSource:0}: Error finding container dc982469c0af36a148cf29aebdcdab78aaff8e4ca4272bc922d4ce1565ca3971: Status 404 returned error can't find the container with id dc982469c0af36a148cf29aebdcdab78aaff8e4ca4272bc922d4ce1565ca3971
Oct 30 13:37:35.089988 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.089941    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:37:35.090076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.090042    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:37:35.090150 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.090090    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:37:35.116038 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.116008    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"]
Oct 30 13:37:35.129491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.129458    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:37:35.129620 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.129548    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:37:35.129620 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.129578    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:37:35.135350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.135315    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss" event={"ID":"262ca860-a303-4568-935a-bb158b4dc3f3","Type":"ContainerDied","Data":"d116fc362f7d85397518df0c3a5b54a51cdd6d1927e215d9b63213903520c491"}
Oct 30 13:37:35.135454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.135389    5893 scope.go:122] "RemoveContainer" containerID="cc89889bf848e635745624ed2b9657ef2f6b3f3cdc9263eb66e96099548f90d7"
Oct 30 13:37:35.135560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.135546    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"
Oct 30 13:37:35.140258 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.140226    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp" event={"ID":"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa","Type":"ContainerStarted","Data":"dc982469c0af36a148cf29aebdcdab78aaff8e4ca4272bc922d4ce1565ca3971"}
Oct 30 13:37:35.148259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.142242    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7" event={"ID":"8ac8d101-9ee3-42fa-9719-66220e201923","Type":"ContainerStarted","Data":"ca4e2499134c7651dea0b9b3e4b1a5a77ca453d892d2348d4dd34107bf434fd0"}
Oct 30 13:37:35.148259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.144632    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"
Oct 30 13:37:35.148259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.144644    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs" event={"ID":"5e392435-687c-4ead-b5c2-44d275dd61d5","Type":"ContainerDied","Data":"ad7288eca0adcda8e6bc9a4c7fab484b4b029a822bb1dd14df76609937eb3bdb"}
Oct 30 13:37:35.148259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.146363    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj" event={"ID":"30fc4668-2aed-4383-b5b1-092a5f3411a5","Type":"ContainerDied","Data":"15e2275736e633c56384dac49362f2f3c758865a38ba375b349c1ee3f1d4e353"}
Oct 30 13:37:35.148259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.146381    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"
Oct 30 13:37:35.164015 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.163993    5893 scope.go:122] "RemoveContainer" containerID="2891664e4348d22eeac18f21e22de7412af541e7c486743763bb12331509628c"
Oct 30 13:37:35.184209 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.184183    5893 scope.go:122] "RemoveContainer" containerID="b39228adeb9377ff748803c785eff1b614622a8d92da47526c198440ead4224c"
Oct 30 13:37:35.215799 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.215764    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"]
Oct 30 13:37:35.225001 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.224865    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-mkmss"]
Oct 30 13:37:35.260815 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.260781    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"]
Oct 30 13:37:35.268092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.267328    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-7gvgj"]
Oct 30 13:37:35.308299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.308178    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"]
Oct 30 13:37:35.310802 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:35.310776    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3c1b28-openshift-default-df56dcb8f-dxwhs"]
Oct 30 13:37:36.208085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.206923    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj" event={"ID":"728d41af-901f-4def-a2c1-aee79df80e1e","Type":"ContainerStarted","Data":"30ad6e481111954c9d55bc109f6d7048332b46fdfecd5ae5bb7fd9ca76069fe8"}
Oct 30 13:37:36.208085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.206994    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj" event={"ID":"728d41af-901f-4def-a2c1-aee79df80e1e","Type":"ContainerStarted","Data":"d28408242921370e865c5f0682f09a7267a92fa86ca30e8e1aad8163fd4ef5b1"}
Oct 30 13:37:36.213434 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.212613    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp" event={"ID":"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa","Type":"ContainerStarted","Data":"b411062db69331b015ac4bb313a3c748070c4004fafd994ff950d237006ee4c2"}
Oct 30 13:37:36.215454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.215336    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7" event={"ID":"8ac8d101-9ee3-42fa-9719-66220e201923","Type":"ContainerStarted","Data":"7d2b1de64dc65e99d9be2293cf6ae4d25954437a5c0e44dc768e8ba5254b162b"}
Oct 30 13:37:36.215626 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.215601    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:36.309411 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.304493    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj" podStartSLOduration=216.304469287 podStartE2EDuration="3m36.304469287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:36.299964196 +0000 UTC m=+3097.883394003" watchObservedRunningTime="2026-10-30 13:37:36.304469287 +0000 UTC m=+3097.887898981"
Oct 30 13:37:36.374300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.359966    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7" podStartSLOduration=161.95948565 podStartE2EDuration="2m42.359945298s" totalImagesPullingTime="400.459648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:36.340238821 +0000 UTC m=+3097.923668515" watchObservedRunningTime="2026-10-30 13:37:36.359945298 +0000 UTC m=+3097.943374992"
Oct 30 13:37:36.406298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.405986    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp" podStartSLOduration=216.405965403 podStartE2EDuration="3m36.405965403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:36.398324547 +0000 UTC m=+3097.981754241" watchObservedRunningTime="2026-10-30 13:37:36.405965403 +0000 UTC m=+3097.989395099"
Oct 30 13:37:36.560311 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.541071    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262ca860-a303-4568-935a-bb158b4dc3f3" path="/var/lib/kubelet/pods/262ca860-a303-4568-935a-bb158b4dc3f3/volumes"
Oct 30 13:37:36.560311 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.542013    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30fc4668-2aed-4383-b5b1-092a5f3411a5" path="/var/lib/kubelet/pods/30fc4668-2aed-4383-b5b1-092a5f3411a5/volumes"
Oct 30 13:37:36.566301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.562042    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e392435-687c-4ead-b5c2-44d275dd61d5" path="/var/lib/kubelet/pods/5e392435-687c-4ead-b5c2-44d275dd61d5/volumes"
Oct 30 13:37:36.639439 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.639407    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:36.644297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.640579    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 30 13:37:36.644297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.640635    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp" podUID="c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 30 13:37:36.705651 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.705599    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:36.709448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.709416    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:37:36.709605 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:36.709539    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj" podUID="728d41af-901f-4def-a2c1-aee79df80e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 13:37:37.638899 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:37.638866    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 30 13:37:37.639380 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:37.639349    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp" podUID="c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 30 13:37:37.715320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:37.714453    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:37:37.715320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:37.714514    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj" podUID="728d41af-901f-4def-a2c1-aee79df80e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 13:37:38.683360 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:38.656796    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 30 13:37:38.685302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:38.685244    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp" podUID="c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 30 13:37:38.721828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:38.711837    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"]
Oct 30 13:37:38.721828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:38.712117    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy" containerID="cri-o://8162d29a76e8454bb10d43803a71764ab59b229649e03d8731b13b1ba206cab8" gracePeriod=30
Oct 30 13:37:38.721828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:38.714807    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:37:38.721828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:38.714851    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj" podUID="728d41af-901f-4def-a2c1-aee79df80e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 13:37:38.737828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:38.725215    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"]
Oct 30 13:37:38.737828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:38.725311    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"]
Oct 30 13:37:38.737828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:38.725510    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8" podUID="b74101d0-b434-47ce-99e5-351977da59ff" containerName="istio-proxy" containerID="cri-o://7fbc76547757f894b1adad2d07c43134239780e9b76e68cab098a39f19c8171e" gracePeriod=30
Oct 30 13:37:38.737828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:38.725823    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9" podUID="5c56fcec-9f01-4323-a89f-ace02d7ac57a" containerName="istio-proxy" containerID="cri-o://01faf263e4099346606082156fb242751b5d5889f66665624bd720d1f576e1c1" gracePeriod=30
Oct 30 13:37:38.998332 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:38.996814    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"
Oct 30 13:37:38.999849 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:38.998639    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"
Oct 30 13:37:39.005593 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:39.005532    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"
Oct 30 13:37:39.731299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:39.724173    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 30 13:37:39.731299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:39.724242    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp" podUID="c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 30 13:37:39.766509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:39.750233    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:37:39.766509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:39.750327    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj" podUID="728d41af-901f-4def-a2c1-aee79df80e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 13:37:40.638915 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:40.638864    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 30 13:37:40.639079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:40.638942    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp" podUID="c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 30 13:37:40.712294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:40.710437    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:37:40.712294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:40.710573    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj" podUID="728d41af-901f-4def-a2c1-aee79df80e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 13:37:41.640115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:41.640082    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Oct 30 13:37:41.666293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:41.666023    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp" podUID="c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Oct 30 13:37:41.705301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:41.702896    5893 patch_prober.go:29] interesting pod/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:37:41.705301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:41.702991    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj" podUID="728d41af-901f-4def-a2c1-aee79df80e1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused"
Oct 30 13:37:42.646138 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:42.646106    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:42.646621 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:42.646551    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:42.647614 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:42.647592    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:37:42.708045 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:42.708019    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:42.708325 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:42.708305    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:42.709147 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:42.709124    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:37:43.245937 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:43.245908    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:37:43.993302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:43.990143    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:37:44.144299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.095803    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-workload-socket\") pod \"b74101d0-b434-47ce-99e5-351977da59ff\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") "
Oct 30 13:37:44.144299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.095854    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-workload-certs\") pod \"b74101d0-b434-47ce-99e5-351977da59ff\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") "
Oct 30 13:37:44.144299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.095915    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b74101d0-b434-47ce-99e5-351977da59ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74101d0-b434-47ce-99e5-351977da59ff-istiod-ca-cert\") pod \"b74101d0-b434-47ce-99e5-351977da59ff\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") "
Oct 30 13:37:44.144299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.095959    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74101d0-b434-47ce-99e5-351977da59ff-kube-api-access-jkzrn\" (UniqueName: \"kubernetes.io/projected/b74101d0-b434-47ce-99e5-351977da59ff-kube-api-access-jkzrn\") pod \"b74101d0-b434-47ce-99e5-351977da59ff\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") "
Oct 30 13:37:44.144299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.096065    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-credential-socket\") pod \"b74101d0-b434-47ce-99e5-351977da59ff\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") "
Oct 30 13:37:44.144299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.096111    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-istio-data\") pod \"b74101d0-b434-47ce-99e5-351977da59ff\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") "
Oct 30 13:37:44.144299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.096146    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b74101d0-b434-47ce-99e5-351977da59ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74101d0-b434-47ce-99e5-351977da59ff-istio-podinfo\") pod \"b74101d0-b434-47ce-99e5-351977da59ff\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") "
Oct 30 13:37:44.144299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.096221    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-istio-envoy\") pod \"b74101d0-b434-47ce-99e5-351977da59ff\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") "
Oct 30 13:37:44.144299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.096606    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74101d0-b434-47ce-99e5-351977da59ff-istio-token\" (UniqueName: \"kubernetes.io/projected/b74101d0-b434-47ce-99e5-351977da59ff-istio-token\") pod \"b74101d0-b434-47ce-99e5-351977da59ff\" (UID: \"b74101d0-b434-47ce-99e5-351977da59ff\") "
Oct 30 13:37:44.172298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.166592    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-workload-certs" pod "b74101d0-b434-47ce-99e5-351977da59ff" (UID: "b74101d0-b434-47ce-99e5-351977da59ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.172298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.166806    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-workload-socket" pod "b74101d0-b434-47ce-99e5-351977da59ff" (UID: "b74101d0-b434-47ce-99e5-351977da59ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.172538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.172414    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-credential-socket" pod "b74101d0-b434-47ce-99e5-351977da59ff" (UID: "b74101d0-b434-47ce-99e5-351977da59ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.172771 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.172743    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74101d0-b434-47ce-99e5-351977da59ff-istiod-ca-cert" pod "b74101d0-b434-47ce-99e5-351977da59ff" (UID: "b74101d0-b434-47ce-99e5-351977da59ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:44.187732 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.187644    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.187732 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.187675    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.187732 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.187690    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b74101d0-b434-47ce-99e5-351977da59ff-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.187732 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.187704    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.272395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.265408    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74101d0-b434-47ce-99e5-351977da59ff-istio-token" pod "b74101d0-b434-47ce-99e5-351977da59ff" (UID: "b74101d0-b434-47ce-99e5-351977da59ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:44.272395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.265514    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74101d0-b434-47ce-99e5-351977da59ff-kube-api-access-jkzrn" pod "b74101d0-b434-47ce-99e5-351977da59ff" (UID: "b74101d0-b434-47ce-99e5-351977da59ff"). InnerVolumeSpecName "kube-api-access-jkzrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:44.272627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.272540    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b74101d0-b434-47ce-99e5-351977da59ff-istio-podinfo" pod "b74101d0-b434-47ce-99e5-351977da59ff" (UID: "b74101d0-b434-47ce-99e5-351977da59ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:37:44.292071 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.280499    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-istio-envoy" pod "b74101d0-b434-47ce-99e5-351977da59ff" (UID: "b74101d0-b434-47ce-99e5-351977da59ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.292071 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.286611    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-istio-data" pod "b74101d0-b434-47ce-99e5-351977da59ff" (UID: "b74101d0-b434-47ce-99e5-351977da59ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.292071 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.289552    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b74101d0-b434-47ce-99e5-351977da59ff-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.292071 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.289581    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkzrn\" (UniqueName: \"kubernetes.io/projected/b74101d0-b434-47ce-99e5-351977da59ff-kube-api-access-jkzrn\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.292071 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.289595    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.292071 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.289609    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b74101d0-b434-47ce-99e5-351977da59ff-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.292071 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.289621    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b74101d0-b434-47ce-99e5-351977da59ff-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.353666 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.352199    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:37:44.395767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.390678    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-envoy\") pod \"84c68bf3-ee6a-498f-aeb1-0318f788488d\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") "
Oct 30 13:37:44.395767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.390893    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-token\" (UniqueName: \"kubernetes.io/projected/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-token\") pod \"84c68bf3-ee6a-498f-aeb1-0318f788488d\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") "
Oct 30 13:37:44.395767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.390962    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-data\") pod \"84c68bf3-ee6a-498f-aeb1-0318f788488d\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") "
Oct 30 13:37:44.395767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.391011    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84c68bf3-ee6a-498f-aeb1-0318f788488d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c68bf3-ee6a-498f-aeb1-0318f788488d-istiod-ca-cert\") pod \"84c68bf3-ee6a-498f-aeb1-0318f788488d\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") "
Oct 30 13:37:44.395767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.391043    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-workload-socket\") pod \"84c68bf3-ee6a-498f-aeb1-0318f788488d\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") "
Oct 30 13:37:44.395767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.391063    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-credential-socket\") pod \"84c68bf3-ee6a-498f-aeb1-0318f788488d\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") "
Oct 30 13:37:44.395767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.391087    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-workload-certs\") pod \"84c68bf3-ee6a-498f-aeb1-0318f788488d\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") "
Oct 30 13:37:44.395767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.391114    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-podinfo\") pod \"84c68bf3-ee6a-498f-aeb1-0318f788488d\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") "
Oct 30 13:37:44.395767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.391131    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84c68bf3-ee6a-498f-aeb1-0318f788488d-kube-api-access-dqq6v\" (UniqueName: \"kubernetes.io/projected/84c68bf3-ee6a-498f-aeb1-0318f788488d-kube-api-access-dqq6v\") pod \"84c68bf3-ee6a-498f-aeb1-0318f788488d\" (UID: \"84c68bf3-ee6a-498f-aeb1-0318f788488d\") "
Oct 30 13:37:44.395767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.392480    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84c68bf3-ee6a-498f-aeb1-0318f788488d-istiod-ca-cert" pod "84c68bf3-ee6a-498f-aeb1-0318f788488d" (UID: "84c68bf3-ee6a-498f-aeb1-0318f788488d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:44.395767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.392742    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-workload-socket" pod "84c68bf3-ee6a-498f-aeb1-0318f788488d" (UID: "84c68bf3-ee6a-498f-aeb1-0318f788488d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.395767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.393041    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-workload-certs" pod "84c68bf3-ee6a-498f-aeb1-0318f788488d" (UID: "84c68bf3-ee6a-498f-aeb1-0318f788488d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.395767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.393097    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-data" pod "84c68bf3-ee6a-498f-aeb1-0318f788488d" (UID: "84c68bf3-ee6a-498f-aeb1-0318f788488d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.395767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.393130    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-credential-socket" pod "84c68bf3-ee6a-498f-aeb1-0318f788488d" (UID: "84c68bf3-ee6a-498f-aeb1-0318f788488d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.396709 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.395992    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-envoy" pod "84c68bf3-ee6a-498f-aeb1-0318f788488d" (UID: "84c68bf3-ee6a-498f-aeb1-0318f788488d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.396709 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.396041    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c68bf3-ee6a-498f-aeb1-0318f788488d-kube-api-access-dqq6v" pod "84c68bf3-ee6a-498f-aeb1-0318f788488d" (UID: "84c68bf3-ee6a-498f-aeb1-0318f788488d"). InnerVolumeSpecName "kube-api-access-dqq6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:44.396709 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.396411    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-podinfo" pod "84c68bf3-ee6a-498f-aeb1-0318f788488d" (UID: "84c68bf3-ee6a-498f-aeb1-0318f788488d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:37:44.396709 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.396623    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-token" pod "84c68bf3-ee6a-498f-aeb1-0318f788488d" (UID: "84c68bf3-ee6a-498f-aeb1-0318f788488d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:44.506333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.502371    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.506333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.502405    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.506333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.502417    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.506333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.502434    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84c68bf3-ee6a-498f-aeb1-0318f788488d-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.506333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.502447    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.506333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.502459    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.506333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.502473    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84c68bf3-ee6a-498f-aeb1-0318f788488d-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.506333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.502489    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84c68bf3-ee6a-498f-aeb1-0318f788488d-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.506333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.502501    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqq6v\" (UniqueName: \"kubernetes.io/projected/84c68bf3-ee6a-498f-aeb1-0318f788488d-kube-api-access-dqq6v\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.525422 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.525359    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:37:44.604027 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.603992    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-envoy\") pod \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") "
Oct 30 13:37:44.604213 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.604055    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c56fcec-9f01-4323-a89f-ace02d7ac57a-kube-api-access-dhkh4\" (UniqueName: \"kubernetes.io/projected/5c56fcec-9f01-4323-a89f-ace02d7ac57a-kube-api-access-dhkh4\") pod \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") "
Oct 30 13:37:44.604213 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.604135    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-credential-socket\") pod \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") "
Oct 30 13:37:44.604213 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.604185    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-data\") pod \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") "
Oct 30 13:37:44.604418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.604293    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-podinfo\") pod \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") "
Oct 30 13:37:44.604481 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.604445    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istiod-ca-cert\") pod \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") "
Oct 30 13:37:44.604534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.604507    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-token\" (UniqueName: \"kubernetes.io/projected/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-token\") pod \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") "
Oct 30 13:37:44.604613 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.604590    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-workload-socket\") pod \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") "
Oct 30 13:37:44.604755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.604704    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-workload-certs\") pod \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\" (UID: \"5c56fcec-9f01-4323-a89f-ace02d7ac57a\") "
Oct 30 13:37:44.606019 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.605980    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-data" pod "5c56fcec-9f01-4323-a89f-ace02d7ac57a" (UID: "5c56fcec-9f01-4323-a89f-ace02d7ac57a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.606129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.606066    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-workload-certs" pod "5c56fcec-9f01-4323-a89f-ace02d7ac57a" (UID: "5c56fcec-9f01-4323-a89f-ace02d7ac57a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.606607 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.606570    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istiod-ca-cert" pod "5c56fcec-9f01-4323-a89f-ace02d7ac57a" (UID: "5c56fcec-9f01-4323-a89f-ace02d7ac57a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:37:44.607180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.607153    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-credential-socket" pod "5c56fcec-9f01-4323-a89f-ace02d7ac57a" (UID: "5c56fcec-9f01-4323-a89f-ace02d7ac57a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.607356 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.607337    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-workload-socket" pod "5c56fcec-9f01-4323-a89f-ace02d7ac57a" (UID: "5c56fcec-9f01-4323-a89f-ace02d7ac57a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.609577 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.609547    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-podinfo" pod "5c56fcec-9f01-4323-a89f-ace02d7ac57a" (UID: "5c56fcec-9f01-4323-a89f-ace02d7ac57a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:37:44.609887 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.609865    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c56fcec-9f01-4323-a89f-ace02d7ac57a-kube-api-access-dhkh4" pod "5c56fcec-9f01-4323-a89f-ace02d7ac57a" (UID: "5c56fcec-9f01-4323-a89f-ace02d7ac57a"). InnerVolumeSpecName "kube-api-access-dhkh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:44.610619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.610585    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-envoy" pod "5c56fcec-9f01-4323-a89f-ace02d7ac57a" (UID: "5c56fcec-9f01-4323-a89f-ace02d7ac57a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:37:44.611968 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.611941    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-token" pod "5c56fcec-9f01-4323-a89f-ace02d7ac57a" (UID: "5c56fcec-9f01-4323-a89f-ace02d7ac57a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:37:44.715025 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.714982    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.715025 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.715023    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhkh4\" (UniqueName: \"kubernetes.io/projected/5c56fcec-9f01-4323-a89f-ace02d7ac57a-kube-api-access-dhkh4\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.715254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.715038    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.715254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.715053    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.715254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.715066    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.715254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.715077    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.715254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.715089    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c56fcec-9f01-4323-a89f-ace02d7ac57a-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.715254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.715103    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.715254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.715114    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c56fcec-9f01-4323-a89f-ace02d7ac57a-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:37:44.760637 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.760601    5893 generic.go:356] "Generic (PLEG): container finished" podID="b74101d0-b434-47ce-99e5-351977da59ff" containerID="7fbc76547757f894b1adad2d07c43134239780e9b76e68cab098a39f19c8171e" exitCode=0
Oct 30 13:37:44.760831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.760672    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"
Oct 30 13:37:44.760831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.760692    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8" event={"ID":"b74101d0-b434-47ce-99e5-351977da59ff","Type":"ContainerDied","Data":"7fbc76547757f894b1adad2d07c43134239780e9b76e68cab098a39f19c8171e"}
Oct 30 13:37:44.760831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.760762    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8" event={"ID":"b74101d0-b434-47ce-99e5-351977da59ff","Type":"ContainerDied","Data":"00fb235ccd3b9e5f538386ed7d49b3b0b1c77f4bb34979da9321312e5b0c058e"}
Oct 30 13:37:44.760831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.760796    5893 scope.go:122] "RemoveContainer" containerID="7fbc76547757f894b1adad2d07c43134239780e9b76e68cab098a39f19c8171e"
Oct 30 13:37:44.763031 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.763007    5893 generic.go:356] "Generic (PLEG): container finished" podID="5c56fcec-9f01-4323-a89f-ace02d7ac57a" containerID="01faf263e4099346606082156fb242751b5d5889f66665624bd720d1f576e1c1" exitCode=0
Oct 30 13:37:44.763154 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.763064    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"
Oct 30 13:37:44.763154 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.763143    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9" event={"ID":"5c56fcec-9f01-4323-a89f-ace02d7ac57a","Type":"ContainerDied","Data":"01faf263e4099346606082156fb242751b5d5889f66665624bd720d1f576e1c1"}
Oct 30 13:37:44.764382 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.763183    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9" event={"ID":"5c56fcec-9f01-4323-a89f-ace02d7ac57a","Type":"ContainerDied","Data":"eecf68c74f20379eeabf0efb49cf7f8e6714c9a5ac9c6834c3090fffe2ab56ca"}
Oct 30 13:37:44.767916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.767146    5893 generic.go:356] "Generic (PLEG): container finished" podID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerID="8162d29a76e8454bb10d43803a71764ab59b229649e03d8731b13b1ba206cab8" exitCode=0
Oct 30 13:37:44.767916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.767338    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" event={"ID":"84c68bf3-ee6a-498f-aeb1-0318f788488d","Type":"ContainerDied","Data":"8162d29a76e8454bb10d43803a71764ab59b229649e03d8731b13b1ba206cab8"}
Oct 30 13:37:44.767916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.767379    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt" event={"ID":"84c68bf3-ee6a-498f-aeb1-0318f788488d","Type":"ContainerDied","Data":"70be24072e078e07c60de020e8404b9962e2c67953fdf6b82bc9f4ff62404fe1"}
Oct 30 13:37:44.767916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.767423    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"
Oct 30 13:37:44.806372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.806180    5893 scope.go:122] "RemoveContainer" containerID="7fbc76547757f894b1adad2d07c43134239780e9b76e68cab098a39f19c8171e"
Oct 30 13:37:44.806590 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:37:44.806569    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbc76547757f894b1adad2d07c43134239780e9b76e68cab098a39f19c8171e\": container with ID starting with 7fbc76547757f894b1adad2d07c43134239780e9b76e68cab098a39f19c8171e not found: ID does not exist" containerID="7fbc76547757f894b1adad2d07c43134239780e9b76e68cab098a39f19c8171e"
Oct 30 13:37:44.806663 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.806605    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbc76547757f894b1adad2d07c43134239780e9b76e68cab098a39f19c8171e"} err="failed to get container status \"7fbc76547757f894b1adad2d07c43134239780e9b76e68cab098a39f19c8171e\": rpc error: code = NotFound desc = could not find container \"7fbc76547757f894b1adad2d07c43134239780e9b76e68cab098a39f19c8171e\": container with ID starting with 7fbc76547757f894b1adad2d07c43134239780e9b76e68cab098a39f19c8171e not found: ID does not exist"
Oct 30 13:37:44.806663 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.806629    5893 scope.go:122] "RemoveContainer" containerID="01faf263e4099346606082156fb242751b5d5889f66665624bd720d1f576e1c1"
Oct 30 13:37:44.817870 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.817850    5893 scope.go:122] "RemoveContainer" containerID="01faf263e4099346606082156fb242751b5d5889f66665624bd720d1f576e1c1"
Oct 30 13:37:44.818163 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:37:44.818144    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01faf263e4099346606082156fb242751b5d5889f66665624bd720d1f576e1c1\": container with ID starting with 01faf263e4099346606082156fb242751b5d5889f66665624bd720d1f576e1c1 not found: ID does not exist" containerID="01faf263e4099346606082156fb242751b5d5889f66665624bd720d1f576e1c1"
Oct 30 13:37:44.818247 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.818169    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01faf263e4099346606082156fb242751b5d5889f66665624bd720d1f576e1c1"} err="failed to get container status \"01faf263e4099346606082156fb242751b5d5889f66665624bd720d1f576e1c1\": rpc error: code = NotFound desc = could not find container \"01faf263e4099346606082156fb242751b5d5889f66665624bd720d1f576e1c1\": container with ID starting with 01faf263e4099346606082156fb242751b5d5889f66665624bd720d1f576e1c1 not found: ID does not exist"
Oct 30 13:37:44.818247 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.818186    5893 scope.go:122] "RemoveContainer" containerID="8162d29a76e8454bb10d43803a71764ab59b229649e03d8731b13b1ba206cab8"
Oct 30 13:37:44.829831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.829816    5893 scope.go:122] "RemoveContainer" containerID="c26c2dff88c0c911ef788fb28fb3c7c1be58d6589d6f694aa062aeaabbc6ddab"
Oct 30 13:37:44.848791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.848768    5893 scope.go:122] "RemoveContainer" containerID="8162d29a76e8454bb10d43803a71764ab59b229649e03d8731b13b1ba206cab8"
Oct 30 13:37:44.857918 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:37:44.849347    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8162d29a76e8454bb10d43803a71764ab59b229649e03d8731b13b1ba206cab8\": container with ID starting with 8162d29a76e8454bb10d43803a71764ab59b229649e03d8731b13b1ba206cab8 not found: ID does not exist" containerID="8162d29a76e8454bb10d43803a71764ab59b229649e03d8731b13b1ba206cab8"
Oct 30 13:37:44.857918 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.849389    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8162d29a76e8454bb10d43803a71764ab59b229649e03d8731b13b1ba206cab8"} err="failed to get container status \"8162d29a76e8454bb10d43803a71764ab59b229649e03d8731b13b1ba206cab8\": rpc error: code = NotFound desc = could not find container \"8162d29a76e8454bb10d43803a71764ab59b229649e03d8731b13b1ba206cab8\": container with ID starting with 8162d29a76e8454bb10d43803a71764ab59b229649e03d8731b13b1ba206cab8 not found: ID does not exist"
Oct 30 13:37:44.857918 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.849418    5893 scope.go:122] "RemoveContainer" containerID="c26c2dff88c0c911ef788fb28fb3c7c1be58d6589d6f694aa062aeaabbc6ddab"
Oct 30 13:37:44.857918 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.857860    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"]
Oct 30 13:37:44.858761 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:37:44.858664    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c26c2dff88c0c911ef788fb28fb3c7c1be58d6589d6f694aa062aeaabbc6ddab\": container with ID starting with c26c2dff88c0c911ef788fb28fb3c7c1be58d6589d6f694aa062aeaabbc6ddab not found: ID does not exist" containerID="c26c2dff88c0c911ef788fb28fb3c7c1be58d6589d6f694aa062aeaabbc6ddab"
Oct 30 13:37:44.858761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.858702    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c26c2dff88c0c911ef788fb28fb3c7c1be58d6589d6f694aa062aeaabbc6ddab"} err="failed to get container status \"c26c2dff88c0c911ef788fb28fb3c7c1be58d6589d6f694aa062aeaabbc6ddab\": rpc error: code = NotFound desc = could not find container \"c26c2dff88c0c911ef788fb28fb3c7c1be58d6589d6f694aa062aeaabbc6ddab\": container with ID starting with c26c2dff88c0c911ef788fb28fb3c7c1be58d6589d6f694aa062aeaabbc6ddab not found: ID does not exist"
Oct 30 13:37:44.870378 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.870342    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-d29jt"]
Oct 30 13:37:44.901566 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.901535    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"]
Oct 30 13:37:44.908174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.908111    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-z7jq8"]
Oct 30 13:37:44.944535 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.944319    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"]
Oct 30 13:37:44.950261 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:44.950235    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-03b5b3-openshift-default-5cf89b677-67cm9"]
Oct 30 13:37:46.536100 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:46.536067    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c56fcec-9f01-4323-a89f-ace02d7ac57a" path="/var/lib/kubelet/pods/5c56fcec-9f01-4323-a89f-ace02d7ac57a/volumes"
Oct 30 13:37:46.536667 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:46.536651    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" path="/var/lib/kubelet/pods/84c68bf3-ee6a-498f-aeb1-0318f788488d/volumes"
Oct 30 13:37:46.537207 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:46.537149    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74101d0-b434-47ce-99e5-351977da59ff" path="/var/lib/kubelet/pods/b74101d0-b434-47ce-99e5-351977da59ff/volumes"
Oct 30 13:37:48.646519 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:37:48.646474    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:37:49.815829 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:49.815794    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"]
Oct 30 13:37:54.816668 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:54.816617    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9" podUID="d9afa66e-f49c-4afb-a25f-928ec672683c" containerName="payload-processing" containerID="cri-o://727d94a13ad8ad2c779a17201d1a5acc91311c430b5c38a996c6bbe78d9f4f95" gracePeriod=25
Oct 30 13:37:56.644465 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:56.644423    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"]
Oct 30 13:37:56.679291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:37:56.679235    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"]
Oct 30 13:38:00.034608 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:00.034571    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9" podUID="d9afa66e-f49c-4afb-a25f-928ec672683c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:38:01.650803 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:01.650733    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv" podUID="914da95d-9ca6-4756-b909-9074526ff434" containerName="payload-processing" containerID="cri-o://2bb7c0d6356441ac57b3f69cc34b253ba1c2500503274ac607ef1be19cfb6fb2" gracePeriod=25
Oct 30 13:38:01.679708 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:01.679649    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj" podUID="8ae38ffb-3bee-4e1d-ad7a-5be201a7771d" containerName="payload-pre-processing" containerID="cri-o://af3df3f8dec993c875a7288571dda4e756e058d94e4a20211e11adcd05f8a66c" gracePeriod=25
Oct 30 13:38:01.939473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:01.939407    5893 generic.go:356] "Generic (PLEG): container finished" podID="914da95d-9ca6-4756-b909-9074526ff434" containerID="2bb7c0d6356441ac57b3f69cc34b253ba1c2500503274ac607ef1be19cfb6fb2" exitCode=0
Oct 30 13:38:01.939710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:01.939584    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv" event={"ID":"914da95d-9ca6-4756-b909-9074526ff434","Type":"ContainerDied","Data":"2bb7c0d6356441ac57b3f69cc34b253ba1c2500503274ac607ef1be19cfb6fb2"}
Oct 30 13:38:01.941696 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:01.941657    5893 generic.go:356] "Generic (PLEG): container finished" podID="8ae38ffb-3bee-4e1d-ad7a-5be201a7771d" containerID="af3df3f8dec993c875a7288571dda4e756e058d94e4a20211e11adcd05f8a66c" exitCode=0
Oct 30 13:38:01.941835 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:01.941712    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj" event={"ID":"8ae38ffb-3bee-4e1d-ad7a-5be201a7771d","Type":"ContainerDied","Data":"af3df3f8dec993c875a7288571dda4e756e058d94e4a20211e11adcd05f8a66c"}
Oct 30 13:38:01.999547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:01.999470    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"
Oct 30 13:38:02.037545 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.037516    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"
Oct 30 13:38:02.088997 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.088957    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/914da95d-9ca6-4756-b909-9074526ff434-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914da95d-9ca6-4756-b909-9074526ff434-plugins-config-volume\") pod \"914da95d-9ca6-4756-b909-9074526ff434\" (UID: \"914da95d-9ca6-4756-b909-9074526ff434\") "
Oct 30 13:38:02.089200 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.089045    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d-kube-api-access-rblx5\" (UniqueName: \"kubernetes.io/projected/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d-kube-api-access-rblx5\") pod \"8ae38ffb-3bee-4e1d-ad7a-5be201a7771d\" (UID: \"8ae38ffb-3bee-4e1d-ad7a-5be201a7771d\") "
Oct 30 13:38:02.089200 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.089145    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/914da95d-9ca6-4756-b909-9074526ff434-kube-api-access-dlhld\" (UniqueName: \"kubernetes.io/projected/914da95d-9ca6-4756-b909-9074526ff434-kube-api-access-dlhld\") pod \"914da95d-9ca6-4756-b909-9074526ff434\" (UID: \"914da95d-9ca6-4756-b909-9074526ff434\") "
Oct 30 13:38:02.089498 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.089475    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d-plugins-config-volume\") pod \"8ae38ffb-3bee-4e1d-ad7a-5be201a7771d\" (UID: \"8ae38ffb-3bee-4e1d-ad7a-5be201a7771d\") "
Oct 30 13:38:02.091067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.091038    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d-plugins-config-volume" pod "8ae38ffb-3bee-4e1d-ad7a-5be201a7771d" (UID: "8ae38ffb-3bee-4e1d-ad7a-5be201a7771d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:02.091556 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.091529    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914da95d-9ca6-4756-b909-9074526ff434-plugins-config-volume" pod "914da95d-9ca6-4756-b909-9074526ff434" (UID: "914da95d-9ca6-4756-b909-9074526ff434"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:02.094788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.094762    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d-kube-api-access-rblx5" pod "8ae38ffb-3bee-4e1d-ad7a-5be201a7771d" (UID: "8ae38ffb-3bee-4e1d-ad7a-5be201a7771d"). InnerVolumeSpecName "kube-api-access-rblx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:02.094968 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.094948    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914da95d-9ca6-4756-b909-9074526ff434-kube-api-access-dlhld" pod "914da95d-9ca6-4756-b909-9074526ff434" (UID: "914da95d-9ca6-4756-b909-9074526ff434"). InnerVolumeSpecName "kube-api-access-dlhld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:02.191349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.191311    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:02.191349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.191338    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/914da95d-9ca6-4756-b909-9074526ff434-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:02.191349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.191348    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rblx5\" (UniqueName: \"kubernetes.io/projected/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d-kube-api-access-rblx5\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:02.191349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.191358    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlhld\" (UniqueName: \"kubernetes.io/projected/914da95d-9ca6-4756-b909-9074526ff434-kube-api-access-dlhld\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:02.952934 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.952844    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"
Oct 30 13:38:02.952934 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.952879    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj" event={"ID":"8ae38ffb-3bee-4e1d-ad7a-5be201a7771d","Type":"ContainerDied","Data":"d1c471b2aafc48b224a625d852d562b09267b31c8738da1660d151ee9fddadec"}
Oct 30 13:38:02.953512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.952943    5893 scope.go:122] "RemoveContainer" containerID="af3df3f8dec993c875a7288571dda4e756e058d94e4a20211e11adcd05f8a66c"
Oct 30 13:38:02.955092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.955064    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv" event={"ID":"914da95d-9ca6-4756-b909-9074526ff434","Type":"ContainerDied","Data":"10cce193c1f11dad2ca7af56f2527405ea06884b12ad479108f1f2c73cf0c56a"}
Oct 30 13:38:02.955234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.955149    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"
Oct 30 13:38:02.972714 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:02.972693    5893 scope.go:122] "RemoveContainer" containerID="2bb7c0d6356441ac57b3f69cc34b253ba1c2500503274ac607ef1be19cfb6fb2"
Oct 30 13:38:03.012300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:03.007075    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"]
Oct 30 13:38:03.021321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:03.017550    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3b47e7-74bbcf9d6c-562nj"]
Oct 30 13:38:03.042841 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:03.042806    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"]
Oct 30 13:38:03.046877 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:03.046850    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3b47e7-7598f7d95b-t5wzv"]
Oct 30 13:38:04.535730 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:04.535698    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ae38ffb-3bee-4e1d-ad7a-5be201a7771d" path="/var/lib/kubelet/pods/8ae38ffb-3bee-4e1d-ad7a-5be201a7771d/volumes"
Oct 30 13:38:04.536137 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:04.536122    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="914da95d-9ca6-4756-b909-9074526ff434" path="/var/lib/kubelet/pods/914da95d-9ca6-4756-b909-9074526ff434/volumes"
Oct 30 13:38:10.033868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:10.033832    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9" podUID="d9afa66e-f49c-4afb-a25f-928ec672683c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:38:15.085773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:15.085682    5893 generic.go:356] "Generic (PLEG): container finished" podID="d9afa66e-f49c-4afb-a25f-928ec672683c" containerID="727d94a13ad8ad2c779a17201d1a5acc91311c430b5c38a996c6bbe78d9f4f95" exitCode=0
Oct 30 13:38:15.086243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:15.085861    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9" event={"ID":"d9afa66e-f49c-4afb-a25f-928ec672683c","Type":"ContainerDied","Data":"727d94a13ad8ad2c779a17201d1a5acc91311c430b5c38a996c6bbe78d9f4f95"}
Oct 30 13:38:15.132784 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:15.132758    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"
Oct 30 13:38:15.309833 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:15.309801    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9afa66e-f49c-4afb-a25f-928ec672683c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9afa66e-f49c-4afb-a25f-928ec672683c-plugins-config-volume\") pod \"d9afa66e-f49c-4afb-a25f-928ec672683c\" (UID: \"d9afa66e-f49c-4afb-a25f-928ec672683c\") "
Oct 30 13:38:15.310121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:15.310098    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9afa66e-f49c-4afb-a25f-928ec672683c-kube-api-access-8mfcj\" (UniqueName: \"kubernetes.io/projected/d9afa66e-f49c-4afb-a25f-928ec672683c-kube-api-access-8mfcj\") pod \"d9afa66e-f49c-4afb-a25f-928ec672683c\" (UID: \"d9afa66e-f49c-4afb-a25f-928ec672683c\") "
Oct 30 13:38:15.310263 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:15.310242    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9afa66e-f49c-4afb-a25f-928ec672683c-plugins-config-volume" pod "d9afa66e-f49c-4afb-a25f-928ec672683c" (UID: "d9afa66e-f49c-4afb-a25f-928ec672683c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:15.310710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:15.310692    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9afa66e-f49c-4afb-a25f-928ec672683c-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:15.312451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:15.312431    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9afa66e-f49c-4afb-a25f-928ec672683c-kube-api-access-8mfcj" pod "d9afa66e-f49c-4afb-a25f-928ec672683c" (UID: "d9afa66e-f49c-4afb-a25f-928ec672683c"). InnerVolumeSpecName "kube-api-access-8mfcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:15.412131 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:15.412040    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mfcj\" (UniqueName: \"kubernetes.io/projected/d9afa66e-f49c-4afb-a25f-928ec672683c-kube-api-access-8mfcj\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:16.095500 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:16.095460    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9" event={"ID":"d9afa66e-f49c-4afb-a25f-928ec672683c","Type":"ContainerDied","Data":"899e8104f88cceb78bd68260186cb0c8219b512e0e50083677af8adadd9674dc"}
Oct 30 13:38:16.095500 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:16.095495    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"
Oct 30 13:38:16.096014 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:16.095529    5893 scope.go:122] "RemoveContainer" containerID="727d94a13ad8ad2c779a17201d1a5acc91311c430b5c38a996c6bbe78d9f4f95"
Oct 30 13:38:16.143389 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:16.143359    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"]
Oct 30 13:38:16.153728 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:16.153695    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-09db16-7488884cbd-x4wp9"]
Oct 30 13:38:16.534736 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:16.534701    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9afa66e-f49c-4afb-a25f-928ec672683c" path="/var/lib/kubelet/pods/d9afa66e-f49c-4afb-a25f-928ec672683c/volumes"
Oct 30 13:38:18.648160 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:38:18.648126    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:38:18.927859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:18.927768    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"]
Oct 30 13:38:18.928037 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:18.927983    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p" podUID="31c056ad-c2a2-4737-b1a6-4808325bb745" containerName="payload-processing" containerID="cri-o://8a1e273ad4a6320470957142c5e501682fbee0054dbbfebce1edad8508cd3593" gracePeriod=30
Oct 30 13:38:18.994490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:18.994458    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"]
Oct 30 13:38:18.995066 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:18.995009    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5" podUID="7716f549-910c-434d-88f9-a36ab4788e3f" containerName="payload-pre-processing" containerID="cri-o://92da90ff4fe1b9e1410e1c3f1b4f4c82132009a7ec3adacbfebbb0e9cc71639f" gracePeriod=30
Oct 30 13:38:19.137206 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.137077    5893 generic.go:356] "Generic (PLEG): container finished" podID="7716f549-910c-434d-88f9-a36ab4788e3f" containerID="92da90ff4fe1b9e1410e1c3f1b4f4c82132009a7ec3adacbfebbb0e9cc71639f" exitCode=0
Oct 30 13:38:19.137206 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.137165    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5" event={"ID":"7716f549-910c-434d-88f9-a36ab4788e3f","Type":"ContainerDied","Data":"92da90ff4fe1b9e1410e1c3f1b4f4c82132009a7ec3adacbfebbb0e9cc71639f"}
Oct 30 13:38:19.139945 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.139633    5893 generic.go:356] "Generic (PLEG): container finished" podID="31c056ad-c2a2-4737-b1a6-4808325bb745" containerID="8a1e273ad4a6320470957142c5e501682fbee0054dbbfebce1edad8508cd3593" exitCode=0
Oct 30 13:38:19.139945 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.139860    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p" event={"ID":"31c056ad-c2a2-4737-b1a6-4808325bb745","Type":"ContainerDied","Data":"8a1e273ad4a6320470957142c5e501682fbee0054dbbfebce1edad8508cd3593"}
Oct 30 13:38:19.577526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.577495    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"
Oct 30 13:38:19.583980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.583953    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"
Oct 30 13:38:19.712372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.712249    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7716f549-910c-434d-88f9-a36ab4788e3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7716f549-910c-434d-88f9-a36ab4788e3f-plugins-config-volume\") pod \"7716f549-910c-434d-88f9-a36ab4788e3f\" (UID: \"7716f549-910c-434d-88f9-a36ab4788e3f\") "
Oct 30 13:38:19.712861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.712540    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7716f549-910c-434d-88f9-a36ab4788e3f-kube-api-access-796p7\" (UniqueName: \"kubernetes.io/projected/7716f549-910c-434d-88f9-a36ab4788e3f-kube-api-access-796p7\") pod \"7716f549-910c-434d-88f9-a36ab4788e3f\" (UID: \"7716f549-910c-434d-88f9-a36ab4788e3f\") "
Oct 30 13:38:19.712861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.712660    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7716f549-910c-434d-88f9-a36ab4788e3f-plugins-config-volume" pod "7716f549-910c-434d-88f9-a36ab4788e3f" (UID: "7716f549-910c-434d-88f9-a36ab4788e3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:19.712861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.712687    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31c056ad-c2a2-4737-b1a6-4808325bb745-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c056ad-c2a2-4737-b1a6-4808325bb745-plugins-config-volume\") pod \"31c056ad-c2a2-4737-b1a6-4808325bb745\" (UID: \"31c056ad-c2a2-4737-b1a6-4808325bb745\") "
Oct 30 13:38:19.712861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.712732    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31c056ad-c2a2-4737-b1a6-4808325bb745-kube-api-access-t7bcb\" (UniqueName: \"kubernetes.io/projected/31c056ad-c2a2-4737-b1a6-4808325bb745-kube-api-access-t7bcb\") pod \"31c056ad-c2a2-4737-b1a6-4808325bb745\" (UID: \"31c056ad-c2a2-4737-b1a6-4808325bb745\") "
Oct 30 13:38:19.713094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.713056    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31c056ad-c2a2-4737-b1a6-4808325bb745-plugins-config-volume" pod "31c056ad-c2a2-4737-b1a6-4808325bb745" (UID: "31c056ad-c2a2-4737-b1a6-4808325bb745"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:19.713512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.713484    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31c056ad-c2a2-4737-b1a6-4808325bb745-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:19.713635 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.713516    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7716f549-910c-434d-88f9-a36ab4788e3f-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:19.714963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.714933    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7716f549-910c-434d-88f9-a36ab4788e3f-kube-api-access-796p7" pod "7716f549-910c-434d-88f9-a36ab4788e3f" (UID: "7716f549-910c-434d-88f9-a36ab4788e3f"). InnerVolumeSpecName "kube-api-access-796p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:19.715155 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.715134    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31c056ad-c2a2-4737-b1a6-4808325bb745-kube-api-access-t7bcb" pod "31c056ad-c2a2-4737-b1a6-4808325bb745" (UID: "31c056ad-c2a2-4737-b1a6-4808325bb745"). InnerVolumeSpecName "kube-api-access-t7bcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:19.744888 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.744858    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"]
Oct 30 13:38:19.745754 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745733    5893 cpu_manager.go:414] "containerMap: removing container" podUID="914da95d-9ca6-4756-b909-9074526ff434" containerName="payload-processing"
Oct 30 13:38:19.745754 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745753    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="914da95d-9ca6-4756-b909-9074526ff434" containerName="payload-processing"
Oct 30 13:38:19.745933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745775    5893 cpu_manager.go:414] "containerMap: removing container" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy"
Oct 30 13:38:19.745933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745782    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy"
Oct 30 13:38:19.745933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745820    5893 cpu_manager.go:414] "containerMap: removing container" podUID="5c56fcec-9f01-4323-a89f-ace02d7ac57a" containerName="istio-proxy"
Oct 30 13:38:19.745933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745828    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c56fcec-9f01-4323-a89f-ace02d7ac57a" containerName="istio-proxy"
Oct 30 13:38:19.745933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745840    5893 cpu_manager.go:414] "containerMap: removing container" podUID="b74101d0-b434-47ce-99e5-351977da59ff" containerName="istio-proxy"
Oct 30 13:38:19.745933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745847    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74101d0-b434-47ce-99e5-351977da59ff" containerName="istio-proxy"
Oct 30 13:38:19.745933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745886    5893 cpu_manager.go:414] "containerMap: removing container" podUID="31c056ad-c2a2-4737-b1a6-4808325bb745" containerName="payload-processing"
Oct 30 13:38:19.745933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745894    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31c056ad-c2a2-4737-b1a6-4808325bb745" containerName="payload-processing"
Oct 30 13:38:19.745933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745911    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d9afa66e-f49c-4afb-a25f-928ec672683c" containerName="payload-processing"
Oct 30 13:38:19.745933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745918    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9afa66e-f49c-4afb-a25f-928ec672683c" containerName="payload-processing"
Oct 30 13:38:19.745933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745931    5893 cpu_manager.go:414] "containerMap: removing container" podUID="7716f549-910c-434d-88f9-a36ab4788e3f" containerName="payload-pre-processing"
Oct 30 13:38:19.746510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745938    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7716f549-910c-434d-88f9-a36ab4788e3f" containerName="payload-pre-processing"
Oct 30 13:38:19.746510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745957    5893 cpu_manager.go:414] "containerMap: removing container" podUID="8ae38ffb-3bee-4e1d-ad7a-5be201a7771d" containerName="payload-pre-processing"
Oct 30 13:38:19.746510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.745966    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ae38ffb-3bee-4e1d-ad7a-5be201a7771d" containerName="payload-pre-processing"
Oct 30 13:38:19.746510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.746165    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="914da95d-9ca6-4756-b909-9074526ff434" containerName="payload-processing"
Oct 30 13:38:19.746510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.746178    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy"
Oct 30 13:38:19.746510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.746194    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy"
Oct 30 13:38:19.746510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.746207    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c56fcec-9f01-4323-a89f-ace02d7ac57a" containerName="istio-proxy"
Oct 30 13:38:19.746510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.746218    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74101d0-b434-47ce-99e5-351977da59ff" containerName="istio-proxy"
Oct 30 13:38:19.746510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.746233    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="31c056ad-c2a2-4737-b1a6-4808325bb745" containerName="payload-processing"
Oct 30 13:38:19.746510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.746244    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9afa66e-f49c-4afb-a25f-928ec672683c" containerName="payload-processing"
Oct 30 13:38:19.746510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.746255    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="7716f549-910c-434d-88f9-a36ab4788e3f" containerName="payload-pre-processing"
Oct 30 13:38:19.746510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.746265    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ae38ffb-3bee-4e1d-ad7a-5be201a7771d" containerName="payload-pre-processing"
Oct 30 13:38:19.750699 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.750680    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:19.774793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.768798    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-ba8145-serving-cert\""
Oct 30 13:38:19.774793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.773521    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"]
Oct 30 13:38:19.814716 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.814679    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:19.814912 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.814838    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt52p\" (UniqueName: \"kubernetes.io/projected/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-kube-api-access-zt52p\") pod \"maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:19.815097 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.815078    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:19.815161 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.815146    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-metrics-certs\") pod \"maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:19.815412 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.815395    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7bcb\" (UniqueName: \"kubernetes.io/projected/31c056ad-c2a2-4737-b1a6-4808325bb745-kube-api-access-t7bcb\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:19.815465 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.815421    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-796p7\" (UniqueName: \"kubernetes.io/projected/7716f549-910c-434d-88f9-a36ab4788e3f-kube-api-access-796p7\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:19.916549 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.916512    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-metrics-certs\") pod \"maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:19.916765 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.916662    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:19.916765 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.916719    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt52p\" (UniqueName: \"kubernetes.io/projected/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-kube-api-access-zt52p\") pod \"maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:19.916913 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.916875    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:19.917549 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.917528    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:19.917635 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:38:19.917573    5893 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-serving-cert: secret "maas-api-e2e-ipp-a-ba8145-serving-cert" not found
Oct 30 13:38:19.917669 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:38:19.917643    5893 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-maas-api-tls podName:47ecad36-94c6-4bf6-8738-a5c19fe9ba78 nodeName:}" failed. No retries permitted until 2026-10-30 13:38:20.417625969 +0000 UTC m=+3142.001055657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-maas-api-tls") pod "maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj" (UID: "47ecad36-94c6-4bf6-8738-a5c19fe9ba78") : secret "maas-api-e2e-ipp-a-ba8145-serving-cert" not found
Oct 30 13:38:19.919247 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.919222    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-metrics-certs\") pod \"maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:19.939132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:19.939098    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt52p\" (UniqueName: \"kubernetes.io/projected/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-kube-api-access-zt52p\") pod \"maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:20.154252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.154223    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"
Oct 30 13:38:20.155397 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.155310    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5" event={"ID":"7716f549-910c-434d-88f9-a36ab4788e3f","Type":"ContainerDied","Data":"61c3b20b9370bad9d69299de54f98b6cc146f01130b2c36cc36d5a5d5a6dbf16"}
Oct 30 13:38:20.155397 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.155377    5893 scope.go:122] "RemoveContainer" containerID="92da90ff4fe1b9e1410e1c3f1b4f4c82132009a7ec3adacbfebbb0e9cc71639f"
Oct 30 13:38:20.159448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.159406    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p" event={"ID":"31c056ad-c2a2-4737-b1a6-4808325bb745","Type":"ContainerDied","Data":"340de6670b1488b3e5c9638d5d4c1aa476147fd351487963b0a542bc72c271bf"}
Oct 30 13:38:20.159621 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.159505    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"
Oct 30 13:38:20.176730 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.176704    5893 scope.go:122] "RemoveContainer" containerID="8a1e273ad4a6320470957142c5e501682fbee0054dbbfebce1edad8508cd3593"
Oct 30 13:38:20.206996 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.206970    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"]
Oct 30 13:38:20.213363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.213340    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4f385-659c7779f5-268f5"]
Oct 30 13:38:20.258319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.258263    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"]
Oct 30 13:38:20.262096 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.262071    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4f385-587868497b-j2p4p"]
Oct 30 13:38:20.425524 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.425447    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:20.428030 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.428011    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-maas-api-tls\") pod \"maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:20.533598 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.533558    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31c056ad-c2a2-4737-b1a6-4808325bb745" path="/var/lib/kubelet/pods/31c056ad-c2a2-4737-b1a6-4808325bb745/volumes"
Oct 30 13:38:20.534111 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.534086    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7716f549-910c-434d-88f9-a36ab4788e3f" path="/var/lib/kubelet/pods/7716f549-910c-434d-88f9-a36ab4788e3f/volumes"
Oct 30 13:38:20.666686 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:20.666657    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:21.040749 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:21.040724    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"]
Oct 30 13:38:21.043626 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:38:21.043593    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ecad36_94c6_4bf6_8738_a5c19fe9ba78.slice/crio-32d8b6c45d7b39ee450624b66206cb9fdaaeaeb3fb1b5927ddd14c0e1e6321ab WatchSource:0}: Error finding container 32d8b6c45d7b39ee450624b66206cb9fdaaeaeb3fb1b5927ddd14c0e1e6321ab: Status 404 returned error can't find the container with id 32d8b6c45d7b39ee450624b66206cb9fdaaeaeb3fb1b5927ddd14c0e1e6321ab
Oct 30 13:38:21.172054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:21.172014    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj" event={"ID":"47ecad36-94c6-4bf6-8738-a5c19fe9ba78","Type":"ContainerStarted","Data":"32d8b6c45d7b39ee450624b66206cb9fdaaeaeb3fb1b5927ddd14c0e1e6321ab"}
Oct 30 13:38:21.962622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:21.960478    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"]
Oct 30 13:38:21.962622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:21.960813    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc" podUID="400e5f9c-8609-4854-af56-2f31915c6f13" containerName="maas-api" containerID="cri-o://339df0ccd3cfd2845eac65a833d72b05ee0a10727d2f49afdd636a453bd31e96" gracePeriod=30
Oct 30 13:38:22.145114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.145073    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"]
Oct 30 13:38:22.145604 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.145332    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df" podUID="020e8b33-8eb0-4a5e-9de7-28b422ef2b62" containerName="payload-processing" containerID="cri-o://9bfe2088090375d09fbecb7057aedd7f731a99e488df6e11e8793ba06d1edc82" gracePeriod=30
Oct 30 13:38:22.212529 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.211180    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj" event={"ID":"47ecad36-94c6-4bf6-8738-a5c19fe9ba78","Type":"ContainerStarted","Data":"1db3a3eb2f9859df346b70cc3c1de3a239d6da220bd9b211b479fd39a173b023"}
Oct 30 13:38:22.212529 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.211335    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:22.213643 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.213617    5893 generic.go:356] "Generic (PLEG): container finished" podID="400e5f9c-8609-4854-af56-2f31915c6f13" containerID="339df0ccd3cfd2845eac65a833d72b05ee0a10727d2f49afdd636a453bd31e96" exitCode=0
Oct 30 13:38:22.213899 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.213655    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc" event={"ID":"400e5f9c-8609-4854-af56-2f31915c6f13","Type":"ContainerDied","Data":"339df0ccd3cfd2845eac65a833d72b05ee0a10727d2f49afdd636a453bd31e96"}
Oct 30 13:38:22.261236 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.261168    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj" podStartSLOduration=2.776952005 podStartE2EDuration="3.26114662s" totalImagesPullingTime="484.194615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:22.259799222 +0000 UTC m=+3143.843228892" watchObservedRunningTime="2026-10-30 13:38:22.26114662 +0000 UTC m=+3143.844576315"
Oct 30 13:38:22.413283 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.413244    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:38:22.564125 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.541790    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400e5f9c-8609-4854-af56-2f31915c6f13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400e5f9c-8609-4854-af56-2f31915c6f13-maas-api-tls\") pod \"400e5f9c-8609-4854-af56-2f31915c6f13\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") "
Oct 30 13:38:22.655328 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.642945    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/400e5f9c-8609-4854-af56-2f31915c6f13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/400e5f9c-8609-4854-af56-2f31915c6f13-metrics-certs\") pod \"400e5f9c-8609-4854-af56-2f31915c6f13\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") "
Oct 30 13:38:22.655328 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.643012    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/400e5f9c-8609-4854-af56-2f31915c6f13-kube-api-access-z4pnf\" (UniqueName: \"kubernetes.io/projected/400e5f9c-8609-4854-af56-2f31915c6f13-kube-api-access-z4pnf\") pod \"400e5f9c-8609-4854-af56-2f31915c6f13\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") "
Oct 30 13:38:22.655328 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.643082    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/400e5f9c-8609-4854-af56-2f31915c6f13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400e5f9c-8609-4854-af56-2f31915c6f13-openshift-service-ca\") pod \"400e5f9c-8609-4854-af56-2f31915c6f13\" (UID: \"400e5f9c-8609-4854-af56-2f31915c6f13\") "
Oct 30 13:38:22.702352 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.702310    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400e5f9c-8609-4854-af56-2f31915c6f13-openshift-service-ca" pod "400e5f9c-8609-4854-af56-2f31915c6f13" (UID: "400e5f9c-8609-4854-af56-2f31915c6f13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:22.705614 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.705585    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400e5f9c-8609-4854-af56-2f31915c6f13-maas-api-tls" pod "400e5f9c-8609-4854-af56-2f31915c6f13" (UID: "400e5f9c-8609-4854-af56-2f31915c6f13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:38:22.721529 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.720961    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"
Oct 30 13:38:22.731980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.730716    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400e5f9c-8609-4854-af56-2f31915c6f13-kube-api-access-z4pnf" pod "400e5f9c-8609-4854-af56-2f31915c6f13" (UID: "400e5f9c-8609-4854-af56-2f31915c6f13"). InnerVolumeSpecName "kube-api-access-z4pnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:22.745912 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.745885    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/020e8b33-8eb0-4a5e-9de7-28b422ef2b62-kube-api-access-gxc8v\" (UniqueName: \"kubernetes.io/projected/020e8b33-8eb0-4a5e-9de7-28b422ef2b62-kube-api-access-gxc8v\") pod \"020e8b33-8eb0-4a5e-9de7-28b422ef2b62\" (UID: \"020e8b33-8eb0-4a5e-9de7-28b422ef2b62\") "
Oct 30 13:38:22.746191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.746172    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/020e8b33-8eb0-4a5e-9de7-28b422ef2b62-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020e8b33-8eb0-4a5e-9de7-28b422ef2b62-plugins-config-volume\") pod \"020e8b33-8eb0-4a5e-9de7-28b422ef2b62\" (UID: \"020e8b33-8eb0-4a5e-9de7-28b422ef2b62\") "
Oct 30 13:38:22.747192 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.747159    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/400e5f9c-8609-4854-af56-2f31915c6f13-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:22.747192 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.747195    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4pnf\" (UniqueName: \"kubernetes.io/projected/400e5f9c-8609-4854-af56-2f31915c6f13-kube-api-access-z4pnf\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:22.747413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.747210    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/400e5f9c-8609-4854-af56-2f31915c6f13-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:22.749784 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.749760    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/020e8b33-8eb0-4a5e-9de7-28b422ef2b62-plugins-config-volume" pod "020e8b33-8eb0-4a5e-9de7-28b422ef2b62" (UID: "020e8b33-8eb0-4a5e-9de7-28b422ef2b62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:22.750352 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.750328    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/020e8b33-8eb0-4a5e-9de7-28b422ef2b62-kube-api-access-gxc8v" pod "020e8b33-8eb0-4a5e-9de7-28b422ef2b62" (UID: "020e8b33-8eb0-4a5e-9de7-28b422ef2b62"). InnerVolumeSpecName "kube-api-access-gxc8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:22.756926 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.756887    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/400e5f9c-8609-4854-af56-2f31915c6f13-metrics-certs" pod "400e5f9c-8609-4854-af56-2f31915c6f13" (UID: "400e5f9c-8609-4854-af56-2f31915c6f13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:38:22.848520 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.848446    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxc8v\" (UniqueName: \"kubernetes.io/projected/020e8b33-8eb0-4a5e-9de7-28b422ef2b62-kube-api-access-gxc8v\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:22.848520 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.848473    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/020e8b33-8eb0-4a5e-9de7-28b422ef2b62-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:22.848520 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:22.848484    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/400e5f9c-8609-4854-af56-2f31915c6f13-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:23.227199 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:23.227167    5893 generic.go:356] "Generic (PLEG): container finished" podID="020e8b33-8eb0-4a5e-9de7-28b422ef2b62" containerID="9bfe2088090375d09fbecb7057aedd7f731a99e488df6e11e8793ba06d1edc82" exitCode=0
Oct 30 13:38:23.227754 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:23.227221    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"
Oct 30 13:38:23.227754 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:23.227469    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df" event={"ID":"020e8b33-8eb0-4a5e-9de7-28b422ef2b62","Type":"ContainerDied","Data":"9bfe2088090375d09fbecb7057aedd7f731a99e488df6e11e8793ba06d1edc82"}
Oct 30 13:38:23.227754 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:23.227522    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df" event={"ID":"020e8b33-8eb0-4a5e-9de7-28b422ef2b62","Type":"ContainerDied","Data":"6e72cada980e18d413061cf36237bc47500dc0ce834a2d72ceee9eaeff1227f0"}
Oct 30 13:38:23.227754 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:23.227620    5893 scope.go:122] "RemoveContainer" containerID="9bfe2088090375d09fbecb7057aedd7f731a99e488df6e11e8793ba06d1edc82"
Oct 30 13:38:23.230572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:23.230553    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"
Oct 30 13:38:23.230817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:23.230562    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc" event={"ID":"400e5f9c-8609-4854-af56-2f31915c6f13","Type":"ContainerDied","Data":"7b452fe774340dd7fafbf3c5ba4d81e1b066f07c1493ae535f2accb511e8767b"}
Oct 30 13:38:23.250004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:23.249978    5893 scope.go:122] "RemoveContainer" containerID="9bfe2088090375d09fbecb7057aedd7f731a99e488df6e11e8793ba06d1edc82"
Oct 30 13:38:23.250455 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:38:23.250422    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bfe2088090375d09fbecb7057aedd7f731a99e488df6e11e8793ba06d1edc82\": container with ID starting with 9bfe2088090375d09fbecb7057aedd7f731a99e488df6e11e8793ba06d1edc82 not found: ID does not exist" containerID="9bfe2088090375d09fbecb7057aedd7f731a99e488df6e11e8793ba06d1edc82"
Oct 30 13:38:23.250551 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:23.250469    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bfe2088090375d09fbecb7057aedd7f731a99e488df6e11e8793ba06d1edc82"} err="failed to get container status \"9bfe2088090375d09fbecb7057aedd7f731a99e488df6e11e8793ba06d1edc82\": rpc error: code = NotFound desc = could not find container \"9bfe2088090375d09fbecb7057aedd7f731a99e488df6e11e8793ba06d1edc82\": container with ID starting with 9bfe2088090375d09fbecb7057aedd7f731a99e488df6e11e8793ba06d1edc82 not found: ID does not exist"
Oct 30 13:38:23.250551 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:23.250494    5893 scope.go:122] "RemoveContainer" containerID="339df0ccd3cfd2845eac65a833d72b05ee0a10727d2f49afdd636a453bd31e96"
Oct 30 13:38:23.289565 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:23.289526    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"]
Oct 30 13:38:23.296319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:23.296292    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-76405c-5496b48fb8-kc8df"]
Oct 30 13:38:23.338452 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:23.338390    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"]
Oct 30 13:38:23.342998 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:23.342973    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-76405c-cbcf8f968-8k2jc"]
Oct 30 13:38:24.534849 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:24.534819    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="020e8b33-8eb0-4a5e-9de7-28b422ef2b62" path="/var/lib/kubelet/pods/020e8b33-8eb0-4a5e-9de7-28b422ef2b62/volumes"
Oct 30 13:38:24.535330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:24.535316    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="400e5f9c-8609-4854-af56-2f31915c6f13" path="/var/lib/kubelet/pods/400e5f9c-8609-4854-af56-2f31915c6f13/volumes"
Oct 30 13:38:28.239317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:28.239253    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:38:34.504312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:34.504255    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"]
Oct 30 13:38:39.505069 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:39.504980    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8" podUID="467105e6-571f-478b-8324-9e61bc980757" containerName="payload-processing" containerID="cri-o://354d364e7c5c4c7c9a8d09992df4c5cc7d51169d6b4900007294922269b3e38f" gracePeriod=25
Oct 30 13:38:39.837572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:39.837538    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"
Oct 30 13:38:39.910526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:39.910490    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/467105e6-571f-478b-8324-9e61bc980757-kube-api-access-k8nxm\" (UniqueName: \"kubernetes.io/projected/467105e6-571f-478b-8324-9e61bc980757-kube-api-access-k8nxm\") pod \"467105e6-571f-478b-8324-9e61bc980757\" (UID: \"467105e6-571f-478b-8324-9e61bc980757\") "
Oct 30 13:38:39.910722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:39.910644    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/467105e6-571f-478b-8324-9e61bc980757-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467105e6-571f-478b-8324-9e61bc980757-plugins-config-volume\") pod \"467105e6-571f-478b-8324-9e61bc980757\" (UID: \"467105e6-571f-478b-8324-9e61bc980757\") "
Oct 30 13:38:39.911320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:39.911289    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/467105e6-571f-478b-8324-9e61bc980757-plugins-config-volume" pod "467105e6-571f-478b-8324-9e61bc980757" (UID: "467105e6-571f-478b-8324-9e61bc980757"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:39.911645 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:39.911621    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/467105e6-571f-478b-8324-9e61bc980757-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:39.913130 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:39.913105    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/467105e6-571f-478b-8324-9e61bc980757-kube-api-access-k8nxm" pod "467105e6-571f-478b-8324-9e61bc980757" (UID: "467105e6-571f-478b-8324-9e61bc980757"). InnerVolumeSpecName "kube-api-access-k8nxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:40.014326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:40.014250    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8nxm\" (UniqueName: \"kubernetes.io/projected/467105e6-571f-478b-8324-9e61bc980757-kube-api-access-k8nxm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:40.401740 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:40.400174    5893 generic.go:356] "Generic (PLEG): container finished" podID="467105e6-571f-478b-8324-9e61bc980757" containerID="354d364e7c5c4c7c9a8d09992df4c5cc7d51169d6b4900007294922269b3e38f" exitCode=0
Oct 30 13:38:40.401740 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:40.400401    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8" event={"ID":"467105e6-571f-478b-8324-9e61bc980757","Type":"ContainerDied","Data":"354d364e7c5c4c7c9a8d09992df4c5cc7d51169d6b4900007294922269b3e38f"}
Oct 30 13:38:40.401740 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:40.400459    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8" event={"ID":"467105e6-571f-478b-8324-9e61bc980757","Type":"ContainerDied","Data":"3ff0b0c38c78df646d0fca2d00bfccd9b5a64fab10038d732764fcec6f8c936b"}
Oct 30 13:38:40.401740 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:40.400489    5893 scope.go:122] "RemoveContainer" containerID="354d364e7c5c4c7c9a8d09992df4c5cc7d51169d6b4900007294922269b3e38f"
Oct 30 13:38:40.401740 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:40.400660    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"
Oct 30 13:38:40.416622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:40.416599    5893 scope.go:122] "RemoveContainer" containerID="354d364e7c5c4c7c9a8d09992df4c5cc7d51169d6b4900007294922269b3e38f"
Oct 30 13:38:40.416919 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:38:40.416896    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"354d364e7c5c4c7c9a8d09992df4c5cc7d51169d6b4900007294922269b3e38f\": container with ID starting with 354d364e7c5c4c7c9a8d09992df4c5cc7d51169d6b4900007294922269b3e38f not found: ID does not exist" containerID="354d364e7c5c4c7c9a8d09992df4c5cc7d51169d6b4900007294922269b3e38f"
Oct 30 13:38:40.416989 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:40.416932    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"354d364e7c5c4c7c9a8d09992df4c5cc7d51169d6b4900007294922269b3e38f"} err="failed to get container status \"354d364e7c5c4c7c9a8d09992df4c5cc7d51169d6b4900007294922269b3e38f\": rpc error: code = NotFound desc = could not find container \"354d364e7c5c4c7c9a8d09992df4c5cc7d51169d6b4900007294922269b3e38f\": container with ID starting with 354d364e7c5c4c7c9a8d09992df4c5cc7d51169d6b4900007294922269b3e38f not found: ID does not exist"
Oct 30 13:38:40.451006 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:40.450973    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"]
Oct 30 13:38:40.458727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:40.458697    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-78543b-7d4fbd8f4b-4r6f8"]
Oct 30 13:38:40.533381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:40.533350    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="467105e6-571f-478b-8324-9e61bc980757" path="/var/lib/kubelet/pods/467105e6-571f-478b-8324-9e61bc980757/volumes"
Oct 30 13:38:45.567169 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:45.567136    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"]
Oct 30 13:38:45.568051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:45.568004    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7" podUID="8ac8d101-9ee3-42fa-9719-66220e201923" containerName="maas-api" containerID="cri-o://7d2b1de64dc65e99d9be2293cf6ae4d25954437a5c0e44dc768e8ba5254b162b" gracePeriod=30
Oct 30 13:38:45.934296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:45.925391    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:38:45.949308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:45.946112    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ac8d101-9ee3-42fa-9719-66220e201923-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ac8d101-9ee3-42fa-9719-66220e201923-maas-api-tls\") pod \"8ac8d101-9ee3-42fa-9719-66220e201923\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") "
Oct 30 13:38:45.949308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:45.946954    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac8d101-9ee3-42fa-9719-66220e201923-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac8d101-9ee3-42fa-9719-66220e201923-openshift-service-ca\") pod \"8ac8d101-9ee3-42fa-9719-66220e201923\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") "
Oct 30 13:38:45.949308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:45.947370    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac8d101-9ee3-42fa-9719-66220e201923-openshift-service-ca" pod "8ac8d101-9ee3-42fa-9719-66220e201923" (UID: "8ac8d101-9ee3-42fa-9719-66220e201923"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:45.949308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:45.947775    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac8d101-9ee3-42fa-9719-66220e201923-kube-api-access-pshmc\" (UniqueName: \"kubernetes.io/projected/8ac8d101-9ee3-42fa-9719-66220e201923-kube-api-access-pshmc\") pod \"8ac8d101-9ee3-42fa-9719-66220e201923\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") "
Oct 30 13:38:45.949308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:45.948788    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ac8d101-9ee3-42fa-9719-66220e201923-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ac8d101-9ee3-42fa-9719-66220e201923-metrics-certs\") pod \"8ac8d101-9ee3-42fa-9719-66220e201923\" (UID: \"8ac8d101-9ee3-42fa-9719-66220e201923\") "
Oct 30 13:38:45.950446 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:45.950166    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ac8d101-9ee3-42fa-9719-66220e201923-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:45.952486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:45.952447    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac8d101-9ee3-42fa-9719-66220e201923-metrics-certs" pod "8ac8d101-9ee3-42fa-9719-66220e201923" (UID: "8ac8d101-9ee3-42fa-9719-66220e201923"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:38:45.953326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:45.953252    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac8d101-9ee3-42fa-9719-66220e201923-maas-api-tls" pod "8ac8d101-9ee3-42fa-9719-66220e201923" (UID: "8ac8d101-9ee3-42fa-9719-66220e201923"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:38:45.956596 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:45.955266    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac8d101-9ee3-42fa-9719-66220e201923-kube-api-access-pshmc" pod "8ac8d101-9ee3-42fa-9719-66220e201923" (UID: "8ac8d101-9ee3-42fa-9719-66220e201923"). InnerVolumeSpecName "kube-api-access-pshmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:46.052214 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:46.052185    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pshmc\" (UniqueName: \"kubernetes.io/projected/8ac8d101-9ee3-42fa-9719-66220e201923-kube-api-access-pshmc\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:46.052214 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:46.052213    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ac8d101-9ee3-42fa-9719-66220e201923-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:46.052467 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:46.052226    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ac8d101-9ee3-42fa-9719-66220e201923-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:46.454631 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:46.454594    5893 generic.go:356] "Generic (PLEG): container finished" podID="8ac8d101-9ee3-42fa-9719-66220e201923" containerID="7d2b1de64dc65e99d9be2293cf6ae4d25954437a5c0e44dc768e8ba5254b162b" exitCode=0
Oct 30 13:38:46.454837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:46.454658    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"
Oct 30 13:38:46.454837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:46.454694    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7" event={"ID":"8ac8d101-9ee3-42fa-9719-66220e201923","Type":"ContainerDied","Data":"7d2b1de64dc65e99d9be2293cf6ae4d25954437a5c0e44dc768e8ba5254b162b"}
Oct 30 13:38:46.454837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:46.454767    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7" event={"ID":"8ac8d101-9ee3-42fa-9719-66220e201923","Type":"ContainerDied","Data":"ca4e2499134c7651dea0b9b3e4b1a5a77ca453d892d2348d4dd34107bf434fd0"}
Oct 30 13:38:46.454837 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:46.454801    5893 scope.go:122] "RemoveContainer" containerID="7d2b1de64dc65e99d9be2293cf6ae4d25954437a5c0e44dc768e8ba5254b162b"
Oct 30 13:38:46.466891 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:46.466870    5893 scope.go:122] "RemoveContainer" containerID="7d2b1de64dc65e99d9be2293cf6ae4d25954437a5c0e44dc768e8ba5254b162b"
Oct 30 13:38:46.467238 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:38:46.467216    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d2b1de64dc65e99d9be2293cf6ae4d25954437a5c0e44dc768e8ba5254b162b\": container with ID starting with 7d2b1de64dc65e99d9be2293cf6ae4d25954437a5c0e44dc768e8ba5254b162b not found: ID does not exist" containerID="7d2b1de64dc65e99d9be2293cf6ae4d25954437a5c0e44dc768e8ba5254b162b"
Oct 30 13:38:46.467403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:46.467244    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d2b1de64dc65e99d9be2293cf6ae4d25954437a5c0e44dc768e8ba5254b162b"} err="failed to get container status \"7d2b1de64dc65e99d9be2293cf6ae4d25954437a5c0e44dc768e8ba5254b162b\": rpc error: code = NotFound desc = could not find container \"7d2b1de64dc65e99d9be2293cf6ae4d25954437a5c0e44dc768e8ba5254b162b\": container with ID starting with 7d2b1de64dc65e99d9be2293cf6ae4d25954437a5c0e44dc768e8ba5254b162b not found: ID does not exist"
Oct 30 13:38:46.511797 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:46.511738    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"]
Oct 30 13:38:46.533066 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:46.533037    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3b47e7-578bf7b645-bfsd7"]
Oct 30 13:38:48.533717 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:48.533685    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac8d101-9ee3-42fa-9719-66220e201923" path="/var/lib/kubelet/pods/8ac8d101-9ee3-42fa-9719-66220e201923/volumes"
Oct 30 13:38:48.646353 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:38:48.646310    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:38:50.667774 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:50.667720    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"]
Oct 30 13:38:50.668401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:50.668000    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4" podUID="bfa3efb6-6744-4a5e-8d30-3c1f23d79b34" containerName="maas-api" containerID="cri-o://ae05b3fe4231bd0f19ab795bb41facfb7304ec99e7f60526d1e871031a8e0d1a" gracePeriod=30
Oct 30 13:38:50.688394 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:50.688351    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4" podUID="bfa3efb6-6744-4a5e-8d30-3c1f23d79b34" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.190:8443/health\": dial tcp 10.134.0.190:8443: connect: connection refused"
Oct 30 13:38:50.813574 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:50.813541    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"]
Oct 30 13:38:50.813811 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:50.813784    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn" podUID="58368e3f-ddb9-4363-9d05-93e5bb0e037e" containerName="payload-processing" containerID="cri-o://4e36f49b911129963f6258c1b6ed7ff13db843a911e7819679f2c3fd64daeacb" gracePeriod=30
Oct 30 13:38:51.057121 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.057080    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:38:51.140464 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.140424    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-maas-api-tls\") pod \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") "
Oct 30 13:38:51.140666 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.140618    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-metrics-certs\") pod \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") "
Oct 30 13:38:51.140745 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.140735    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-openshift-service-ca\") pod \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") "
Oct 30 13:38:51.140804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.140788    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-kube-api-access-kss59\" (UniqueName: \"kubernetes.io/projected/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-kube-api-access-kss59\") pod \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\" (UID: \"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34\") "
Oct 30 13:38:51.146642 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.143243    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-openshift-service-ca" pod "bfa3efb6-6744-4a5e-8d30-3c1f23d79b34" (UID: "bfa3efb6-6744-4a5e-8d30-3c1f23d79b34"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:51.146642 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.143897    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:51.150755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.150290    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-metrics-certs" pod "bfa3efb6-6744-4a5e-8d30-3c1f23d79b34" (UID: "bfa3efb6-6744-4a5e-8d30-3c1f23d79b34"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:38:51.155464 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.154795    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-maas-api-tls" pod "bfa3efb6-6744-4a5e-8d30-3c1f23d79b34" (UID: "bfa3efb6-6744-4a5e-8d30-3c1f23d79b34"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:38:51.179768 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.179503    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-kube-api-access-kss59" pod "bfa3efb6-6744-4a5e-8d30-3c1f23d79b34" (UID: "bfa3efb6-6744-4a5e-8d30-3c1f23d79b34"). InnerVolumeSpecName "kube-api-access-kss59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:51.194483 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.194459    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"
Oct 30 13:38:51.282299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.279504    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58368e3f-ddb9-4363-9d05-93e5bb0e037e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58368e3f-ddb9-4363-9d05-93e5bb0e037e-plugins-config-volume\") pod \"58368e3f-ddb9-4363-9d05-93e5bb0e037e\" (UID: \"58368e3f-ddb9-4363-9d05-93e5bb0e037e\") "
Oct 30 13:38:51.282299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.279643    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58368e3f-ddb9-4363-9d05-93e5bb0e037e-kube-api-access-2lbkd\" (UniqueName: \"kubernetes.io/projected/58368e3f-ddb9-4363-9d05-93e5bb0e037e-kube-api-access-2lbkd\") pod \"58368e3f-ddb9-4363-9d05-93e5bb0e037e\" (UID: \"58368e3f-ddb9-4363-9d05-93e5bb0e037e\") "
Oct 30 13:38:51.282299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.280417    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:51.282299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.280443    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kss59\" (UniqueName: \"kubernetes.io/projected/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-kube-api-access-kss59\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:51.309384 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.291324    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:51.347344 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.323436    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58368e3f-ddb9-4363-9d05-93e5bb0e037e-plugins-config-volume" pod "58368e3f-ddb9-4363-9d05-93e5bb0e037e" (UID: "58368e3f-ddb9-4363-9d05-93e5bb0e037e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:51.451297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.451243    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58368e3f-ddb9-4363-9d05-93e5bb0e037e-kube-api-access-2lbkd" pod "58368e3f-ddb9-4363-9d05-93e5bb0e037e" (UID: "58368e3f-ddb9-4363-9d05-93e5bb0e037e"). InnerVolumeSpecName "kube-api-access-2lbkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:51.460527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.458746    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58368e3f-ddb9-4363-9d05-93e5bb0e037e-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:51.580654 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.580571    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lbkd\" (UniqueName: \"kubernetes.io/projected/58368e3f-ddb9-4363-9d05-93e5bb0e037e-kube-api-access-2lbkd\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:38:51.583909 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.583877    5893 generic.go:356] "Generic (PLEG): container finished" podID="58368e3f-ddb9-4363-9d05-93e5bb0e037e" containerID="4e36f49b911129963f6258c1b6ed7ff13db843a911e7819679f2c3fd64daeacb" exitCode=0
Oct 30 13:38:51.584092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.584004    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn" event={"ID":"58368e3f-ddb9-4363-9d05-93e5bb0e037e","Type":"ContainerDied","Data":"4e36f49b911129963f6258c1b6ed7ff13db843a911e7819679f2c3fd64daeacb"}
Oct 30 13:38:51.584092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.584057    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn" event={"ID":"58368e3f-ddb9-4363-9d05-93e5bb0e037e","Type":"ContainerDied","Data":"165e7dad2873f5a9d19f90be8d5ae3aec7d20a8ac60075290cf96ee9c8d06db4"}
Oct 30 13:38:51.584092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.584086    5893 scope.go:122] "RemoveContainer" containerID="4e36f49b911129963f6258c1b6ed7ff13db843a911e7819679f2c3fd64daeacb"
Oct 30 13:38:51.584325 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.584232    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"
Oct 30 13:38:51.604331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.604303    5893 generic.go:356] "Generic (PLEG): container finished" podID="bfa3efb6-6744-4a5e-8d30-3c1f23d79b34" containerID="ae05b3fe4231bd0f19ab795bb41facfb7304ec99e7f60526d1e871031a8e0d1a" exitCode=0
Oct 30 13:38:51.604819 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.604796    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4" event={"ID":"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34","Type":"ContainerDied","Data":"ae05b3fe4231bd0f19ab795bb41facfb7304ec99e7f60526d1e871031a8e0d1a"}
Oct 30 13:38:51.605025 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.604995    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4" event={"ID":"bfa3efb6-6744-4a5e-8d30-3c1f23d79b34","Type":"ContainerDied","Data":"bccac22a4af094e7f341de9271910388d0bafe43225a904229ad1bc9cf2322bc"}
Oct 30 13:38:51.605301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.605260    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"
Oct 30 13:38:51.620945 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.620862    5893 scope.go:122] "RemoveContainer" containerID="4e36f49b911129963f6258c1b6ed7ff13db843a911e7819679f2c3fd64daeacb"
Oct 30 13:38:51.621380 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:38:51.621351    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e36f49b911129963f6258c1b6ed7ff13db843a911e7819679f2c3fd64daeacb\": container with ID starting with 4e36f49b911129963f6258c1b6ed7ff13db843a911e7819679f2c3fd64daeacb not found: ID does not exist" containerID="4e36f49b911129963f6258c1b6ed7ff13db843a911e7819679f2c3fd64daeacb"
Oct 30 13:38:51.621554 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.621530    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e36f49b911129963f6258c1b6ed7ff13db843a911e7819679f2c3fd64daeacb"} err="failed to get container status \"4e36f49b911129963f6258c1b6ed7ff13db843a911e7819679f2c3fd64daeacb\": rpc error: code = NotFound desc = could not find container \"4e36f49b911129963f6258c1b6ed7ff13db843a911e7819679f2c3fd64daeacb\": container with ID starting with 4e36f49b911129963f6258c1b6ed7ff13db843a911e7819679f2c3fd64daeacb not found: ID does not exist"
Oct 30 13:38:51.621662 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.621648    5893 scope.go:122] "RemoveContainer" containerID="ae05b3fe4231bd0f19ab795bb41facfb7304ec99e7f60526d1e871031a8e0d1a"
Oct 30 13:38:51.643881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.643853    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"]
Oct 30 13:38:51.649850 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.649771    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1c8130-f779f66bd-24qfn"]
Oct 30 13:38:51.659381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.651300    5893 scope.go:122] "RemoveContainer" containerID="ae05b3fe4231bd0f19ab795bb41facfb7304ec99e7f60526d1e871031a8e0d1a"
Oct 30 13:38:51.659381 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:38:51.651707    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae05b3fe4231bd0f19ab795bb41facfb7304ec99e7f60526d1e871031a8e0d1a\": container with ID starting with ae05b3fe4231bd0f19ab795bb41facfb7304ec99e7f60526d1e871031a8e0d1a not found: ID does not exist" containerID="ae05b3fe4231bd0f19ab795bb41facfb7304ec99e7f60526d1e871031a8e0d1a"
Oct 30 13:38:51.659381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.651741    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae05b3fe4231bd0f19ab795bb41facfb7304ec99e7f60526d1e871031a8e0d1a"} err="failed to get container status \"ae05b3fe4231bd0f19ab795bb41facfb7304ec99e7f60526d1e871031a8e0d1a\": rpc error: code = NotFound desc = could not find container \"ae05b3fe4231bd0f19ab795bb41facfb7304ec99e7f60526d1e871031a8e0d1a\": container with ID starting with ae05b3fe4231bd0f19ab795bb41facfb7304ec99e7f60526d1e871031a8e0d1a not found: ID does not exist"
Oct 30 13:38:51.737516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.724645    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"]
Oct 30 13:38:51.793696 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:51.793656    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1c8130-7c4566cc8b-49mk4"]
Oct 30 13:38:52.533750 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:52.533719    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58368e3f-ddb9-4363-9d05-93e5bb0e037e" path="/var/lib/kubelet/pods/58368e3f-ddb9-4363-9d05-93e5bb0e037e/volumes"
Oct 30 13:38:52.534364 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:52.534342    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa3efb6-6744-4a5e-8d30-3c1f23d79b34" path="/var/lib/kubelet/pods/bfa3efb6-6744-4a5e-8d30-3c1f23d79b34/volumes"
Oct 30 13:38:54.694567 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:54.694513    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"]
Oct 30 13:38:54.695067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:54.694736    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh" podUID="895b7a55-b271-4b61-a3c5-7411e8b9a9b0" containerName="istio-proxy" containerID="cri-o://87554b331c25f5c68c62eadcb8571a70cc0c5c059b6a8a250c0baee48a992e57" gracePeriod=30
Oct 30 13:38:54.936294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:54.930843    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"]
Oct 30 13:38:54.936294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:54.931212    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg" podUID="5e678e9e-ec06-4b9e-b1f6-cebb61e59571" containerName="istio-proxy" containerID="cri-o://d1deecbc92d7224dc05cdb0e0940518baa5171fb41841eae190fe62cb1cf5553" gracePeriod=30
Oct 30 13:38:54.970450 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:54.970415    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"]
Oct 30 13:38:54.970752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:54.970725    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w" podUID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerName="istio-proxy" containerID="cri-o://457d9b6276efd816731ecf5561cb5fbe208cd49b166648cfb1745fe8575429ff" gracePeriod=30
Oct 30 13:38:55.405055 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:55.404954    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"]
Oct 30 13:38:55.405324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:55.405247    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws" podUID="f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" containerName="istio-proxy" containerID="cri-o://5da8786df647cbdbd6f953bc7ac7d16aa57775703b07ada147a2dabeb0c94961" gracePeriod=30
Oct 30 13:38:55.913786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:55.913752    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"]
Oct 30 13:38:55.978298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:55.975114    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"]
Oct 30 13:38:58.868317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:58.867805    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"]
Oct 30 13:38:58.868317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:58.868133    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp" podUID="c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" containerName="istio-proxy" containerID="cri-o://b411062db69331b015ac4bb313a3c748070c4004fafd994ff950d237006ee4c2" gracePeriod=30
Oct 30 13:38:58.880180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:58.880132    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"]
Oct 30 13:38:58.880534 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:58.880505    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj" podUID="728d41af-901f-4def-a2c1-aee79df80e1e" containerName="istio-proxy" containerID="cri-o://30ad6e481111954c9d55bc109f6d7048332b46fdfecd5ae5bb7fd9ca76069fe8" gracePeriod=30
Oct 30 13:38:58.887245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:58.887208    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"]
Oct 30 13:38:58.887548 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:58.887502    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerName="istio-proxy" containerID="cri-o://43ec74448547cda39d22fbe5e3d1b0b1bc4e335bf70251a9bcb78a7f7170cf96" gracePeriod=30
Oct 30 13:38:59.659255 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:59.659213    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:38:59.659605 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:59.659289    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg" podUID="5e678e9e-ec06-4b9e-b1f6-cebb61e59571" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:38:59.682063 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:59.682027    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:38:59.682230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:38:59.682090    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh" podUID="895b7a55-b271-4b61-a3c5-7411e8b9a9b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:39:00.142539 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.142511    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:39:00.235805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.235698    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"]
Oct 30 13:39:00.236549 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.236325    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln" podUID="8525f4ea-caea-4321-8c2e-4c6fa4fe06da" containerName="istio-proxy" containerID="cri-o://610b534d7f749ae4444f821a6513c40fdbb0a774cd3e09093c088aac338228f7" gracePeriod=30
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.256739    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istiod-ca-cert\") pod \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") "
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.256803    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-credential-socket\") pod \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") "
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.256849    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-token\" (UniqueName: \"kubernetes.io/projected/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-token\") pod \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") "
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.256968    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-workload-certs\") pod \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") "
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.257000    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-workload-socket\") pod \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") "
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.257023    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-podinfo\") pod \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") "
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.257107    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-kube-api-access-dj9j9\" (UniqueName: \"kubernetes.io/projected/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-kube-api-access-dj9j9\") pod \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") "
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.257198    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-envoy\") pod \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") "
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.257218    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-data\") pod \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\" (UID: \"895b7a55-b271-4b61-a3c5-7411e8b9a9b0\") "
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.258462    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-data" pod "895b7a55-b271-4b61-a3c5-7411e8b9a9b0" (UID: "895b7a55-b271-4b61-a3c5-7411e8b9a9b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.258652    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-credential-socket" pod "895b7a55-b271-4b61-a3c5-7411e8b9a9b0" (UID: "895b7a55-b271-4b61-a3c5-7411e8b9a9b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.258766    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-workload-certs" pod "895b7a55-b271-4b61-a3c5-7411e8b9a9b0" (UID: "895b7a55-b271-4b61-a3c5-7411e8b9a9b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.258847    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-workload-socket" pod "895b7a55-b271-4b61-a3c5-7411e8b9a9b0" (UID: "895b7a55-b271-4b61-a3c5-7411e8b9a9b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.258989    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istiod-ca-cert" pod "895b7a55-b271-4b61-a3c5-7411e8b9a9b0" (UID: "895b7a55-b271-4b61-a3c5-7411e8b9a9b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.262497    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-kube-api-access-dj9j9" pod "895b7a55-b271-4b61-a3c5-7411e8b9a9b0" (UID: "895b7a55-b271-4b61-a3c5-7411e8b9a9b0"). InnerVolumeSpecName "kube-api-access-dj9j9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.265294 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.262947    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-envoy" pod "895b7a55-b271-4b61-a3c5-7411e8b9a9b0" (UID: "895b7a55-b271-4b61-a3c5-7411e8b9a9b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.266345 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.265606    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-token" pod "895b7a55-b271-4b61-a3c5-7411e8b9a9b0" (UID: "895b7a55-b271-4b61-a3c5-7411e8b9a9b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.287137 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.266809    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-podinfo" pod "895b7a55-b271-4b61-a3c5-7411e8b9a9b0" (UID: "895b7a55-b271-4b61-a3c5-7411e8b9a9b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:00.306858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.306754    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:39:00.365298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.364632    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-credential-socket\") pod \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") "
Oct 30 13:39:00.365298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.364702    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-workload-socket\") pod \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") "
Oct 30 13:39:00.365298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.364827    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-envoy\") pod \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") "
Oct 30 13:39:00.365298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.364873    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-workload-certs\") pod \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") "
Oct 30 13:39:00.365298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.364903    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-data\") pod \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") "
Oct 30 13:39:00.365298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.365033    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-podinfo\") pod \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") "
Oct 30 13:39:00.365298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.365124    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-token\" (UniqueName: \"kubernetes.io/projected/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-token\") pod \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") "
Oct 30 13:39:00.365298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.365176    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-kube-api-access-ggjk5\" (UniqueName: \"kubernetes.io/projected/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-kube-api-access-ggjk5\") pod \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") "
Oct 30 13:39:00.365298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.365267    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istiod-ca-cert\") pod \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\" (UID: \"5e678e9e-ec06-4b9e-b1f6-cebb61e59571\") "
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.366096    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.366117    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.366127    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.366136    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj9j9\" (UniqueName: \"kubernetes.io/projected/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-kube-api-access-dj9j9\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.366145    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.366153    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.366160    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.366168    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.366175    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/895b7a55-b271-4b61-a3c5-7411e8b9a9b0-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.368460    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-credential-socket" pod "5e678e9e-ec06-4b9e-b1f6-cebb61e59571" (UID: "5e678e9e-ec06-4b9e-b1f6-cebb61e59571"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.368765    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-data" pod "5e678e9e-ec06-4b9e-b1f6-cebb61e59571" (UID: "5e678e9e-ec06-4b9e-b1f6-cebb61e59571"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.369030    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-workload-socket" pod "5e678e9e-ec06-4b9e-b1f6-cebb61e59571" (UID: "5e678e9e-ec06-4b9e-b1f6-cebb61e59571"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.369085    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-workload-certs" pod "5e678e9e-ec06-4b9e-b1f6-cebb61e59571" (UID: "5e678e9e-ec06-4b9e-b1f6-cebb61e59571"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.369220    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istiod-ca-cert" pod "5e678e9e-ec06-4b9e-b1f6-cebb61e59571" (UID: "5e678e9e-ec06-4b9e-b1f6-cebb61e59571"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:00.381371 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.375571    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-kube-api-access-ggjk5" pod "5e678e9e-ec06-4b9e-b1f6-cebb61e59571" (UID: "5e678e9e-ec06-4b9e-b1f6-cebb61e59571"). InnerVolumeSpecName "kube-api-access-ggjk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.382452 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.382407    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-podinfo" pod "5e678e9e-ec06-4b9e-b1f6-cebb61e59571" (UID: "5e678e9e-ec06-4b9e-b1f6-cebb61e59571"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:00.382698 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.382663    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-envoy" pod "5e678e9e-ec06-4b9e-b1f6-cebb61e59571" (UID: "5e678e9e-ec06-4b9e-b1f6-cebb61e59571"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.383831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.383795    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-token" pod "5e678e9e-ec06-4b9e-b1f6-cebb61e59571" (UID: "5e678e9e-ec06-4b9e-b1f6-cebb61e59571"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.479230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.479053    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.479230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.479089    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.479230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.479108    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.479230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.479125    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.479230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.479142    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.479230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.479155    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.479230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.479169    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.479230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.479181    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.479230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.479196    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggjk5\" (UniqueName: \"kubernetes.io/projected/5e678e9e-ec06-4b9e-b1f6-cebb61e59571-kube-api-access-ggjk5\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.770572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.769064    5893 generic.go:356] "Generic (PLEG): container finished" podID="895b7a55-b271-4b61-a3c5-7411e8b9a9b0" containerID="87554b331c25f5c68c62eadcb8571a70cc0c5c059b6a8a250c0baee48a992e57" exitCode=0
Oct 30 13:39:00.770572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.769217    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh" event={"ID":"895b7a55-b271-4b61-a3c5-7411e8b9a9b0","Type":"ContainerDied","Data":"87554b331c25f5c68c62eadcb8571a70cc0c5c059b6a8a250c0baee48a992e57"}
Oct 30 13:39:00.770572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.769291    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh" event={"ID":"895b7a55-b271-4b61-a3c5-7411e8b9a9b0","Type":"ContainerDied","Data":"d9ef5342e6b0205ff79465db0380407d5fecc1610fe5a18cb084f5bed3b2979b"}
Oct 30 13:39:00.770572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.769329    5893 scope.go:122] "RemoveContainer" containerID="87554b331c25f5c68c62eadcb8571a70cc0c5c059b6a8a250c0baee48a992e57"
Oct 30 13:39:00.770572 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.769481    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"
Oct 30 13:39:00.777169 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.775870    5893 generic.go:356] "Generic (PLEG): container finished" podID="5e678e9e-ec06-4b9e-b1f6-cebb61e59571" containerID="d1deecbc92d7224dc05cdb0e0940518baa5171fb41841eae190fe62cb1cf5553" exitCode=0
Oct 30 13:39:00.777169 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.776023    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg" event={"ID":"5e678e9e-ec06-4b9e-b1f6-cebb61e59571","Type":"ContainerDied","Data":"d1deecbc92d7224dc05cdb0e0940518baa5171fb41841eae190fe62cb1cf5553"}
Oct 30 13:39:00.777169 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.776074    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg" event={"ID":"5e678e9e-ec06-4b9e-b1f6-cebb61e59571","Type":"ContainerDied","Data":"a712adca07201779a0f01a54e681474eba632d7258ca8b33c5b3034949e6a4be"}
Oct 30 13:39:00.777169 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.776157    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"
Oct 30 13:39:00.779715 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.779449    5893 generic.go:356] "Generic (PLEG): container finished" podID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerID="457d9b6276efd816731ecf5561cb5fbe208cd49b166648cfb1745fe8575429ff" exitCode=0
Oct 30 13:39:00.779715 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.779575    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w" event={"ID":"45a767a0-a88a-4a84-b0e1-c918badbd3e9","Type":"ContainerDied","Data":"457d9b6276efd816731ecf5561cb5fbe208cd49b166648cfb1745fe8575429ff"}
Oct 30 13:39:00.779715 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.779621    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w" event={"ID":"45a767a0-a88a-4a84-b0e1-c918badbd3e9","Type":"ContainerDied","Data":"5a340da65b3b0fbff5ec7221e51e1809241a8d4bf88811fc7d18419ac058c0ae"}
Oct 30 13:39:00.779715 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.779646    5893 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a340da65b3b0fbff5ec7221e51e1809241a8d4bf88811fc7d18419ac058c0ae"
Oct 30 13:39:00.783816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.783795    5893 generic.go:356] "Generic (PLEG): container finished" podID="f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" containerID="5da8786df647cbdbd6f953bc7ac7d16aa57775703b07ada147a2dabeb0c94961" exitCode=0
Oct 30 13:39:00.784104 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.783965    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws" event={"ID":"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d","Type":"ContainerDied","Data":"5da8786df647cbdbd6f953bc7ac7d16aa57775703b07ada147a2dabeb0c94961"}
Oct 30 13:39:00.797817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.797793    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:39:00.819834 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.819224    5893 scope.go:122] "RemoveContainer" containerID="87554b331c25f5c68c62eadcb8571a70cc0c5c059b6a8a250c0baee48a992e57"
Oct 30 13:39:00.819834 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:00.819640    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87554b331c25f5c68c62eadcb8571a70cc0c5c059b6a8a250c0baee48a992e57\": container with ID starting with 87554b331c25f5c68c62eadcb8571a70cc0c5c059b6a8a250c0baee48a992e57 not found: ID does not exist" containerID="87554b331c25f5c68c62eadcb8571a70cc0c5c059b6a8a250c0baee48a992e57"
Oct 30 13:39:00.819834 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.819679    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87554b331c25f5c68c62eadcb8571a70cc0c5c059b6a8a250c0baee48a992e57"} err="failed to get container status \"87554b331c25f5c68c62eadcb8571a70cc0c5c059b6a8a250c0baee48a992e57\": rpc error: code = NotFound desc = could not find container \"87554b331c25f5c68c62eadcb8571a70cc0c5c059b6a8a250c0baee48a992e57\": container with ID starting with 87554b331c25f5c68c62eadcb8571a70cc0c5c059b6a8a250c0baee48a992e57 not found: ID does not exist"
Oct 30 13:39:00.819834 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.819697    5893 scope.go:122] "RemoveContainer" containerID="d1deecbc92d7224dc05cdb0e0940518baa5171fb41841eae190fe62cb1cf5553"
Oct 30 13:39:00.837620 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.837590    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"]
Oct 30 13:39:00.847310 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.847260    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-pm6vg"]
Oct 30 13:39:00.863351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.863322    5893 scope.go:122] "RemoveContainer" containerID="d1deecbc92d7224dc05cdb0e0940518baa5171fb41841eae190fe62cb1cf5553"
Oct 30 13:39:00.863816 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:00.863794    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1deecbc92d7224dc05cdb0e0940518baa5171fb41841eae190fe62cb1cf5553\": container with ID starting with d1deecbc92d7224dc05cdb0e0940518baa5171fb41841eae190fe62cb1cf5553 not found: ID does not exist" containerID="d1deecbc92d7224dc05cdb0e0940518baa5171fb41841eae190fe62cb1cf5553"
Oct 30 13:39:00.863986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.863960    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1deecbc92d7224dc05cdb0e0940518baa5171fb41841eae190fe62cb1cf5553"} err="failed to get container status \"d1deecbc92d7224dc05cdb0e0940518baa5171fb41841eae190fe62cb1cf5553\": rpc error: code = NotFound desc = could not find container \"d1deecbc92d7224dc05cdb0e0940518baa5171fb41841eae190fe62cb1cf5553\": container with ID starting with d1deecbc92d7224dc05cdb0e0940518baa5171fb41841eae190fe62cb1cf5553 not found: ID does not exist"
Oct 30 13:39:00.863986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.863986    5893 scope.go:122] "RemoveContainer" containerID="1484fa554faadcf420790790e30c2bad1efaf2cd2a5d01c1b45044a8cb42731f"
Oct 30 13:39:00.866898 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.866871    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:39:00.879524 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.879026    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"]
Oct 30 13:39:00.885546 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.885514    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-vlwfh"]
Oct 30 13:39:00.893183 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.893156    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a767a0-a88a-4a84-b0e1-c918badbd3e9-kube-api-access-mnt6m\" (UniqueName: \"kubernetes.io/projected/45a767a0-a88a-4a84-b0e1-c918badbd3e9-kube-api-access-mnt6m\") pod \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") "
Oct 30 13:39:00.893369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.893314    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-token\" (UniqueName: \"kubernetes.io/projected/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-token\") pod \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") "
Oct 30 13:39:00.893369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.893360    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-workload-certs\") pod \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") "
Oct 30 13:39:00.893505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.893387    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-credential-socket\") pod \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") "
Oct 30 13:39:00.893505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.893448    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-workload-socket\") pod \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") "
Oct 30 13:39:00.893623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.893565    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istiod-ca-cert\") pod \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") "
Oct 30 13:39:00.893679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.893625    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-podinfo\") pod \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") "
Oct 30 13:39:00.893679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.893654    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-envoy\") pod \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") "
Oct 30 13:39:00.893679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.893657    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-workload-certs" pod "45a767a0-a88a-4a84-b0e1-c918badbd3e9" (UID: "45a767a0-a88a-4a84-b0e1-c918badbd3e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.893940 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.893713    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-data\") pod \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\" (UID: \"45a767a0-a88a-4a84-b0e1-c918badbd3e9\") "
Oct 30 13:39:00.893940 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.893721    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-credential-socket" pod "45a767a0-a88a-4a84-b0e1-c918badbd3e9" (UID: "45a767a0-a88a-4a84-b0e1-c918badbd3e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.893940 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.893934    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-data" pod "45a767a0-a88a-4a84-b0e1-c918badbd3e9" (UID: "45a767a0-a88a-4a84-b0e1-c918badbd3e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.894379 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.894351    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-workload-socket" pod "45a767a0-a88a-4a84-b0e1-c918badbd3e9" (UID: "45a767a0-a88a-4a84-b0e1-c918badbd3e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.894509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.894450    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.894509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.894466    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.894509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.894479    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:00.894676 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.894606    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istiod-ca-cert" pod "45a767a0-a88a-4a84-b0e1-c918badbd3e9" (UID: "45a767a0-a88a-4a84-b0e1-c918badbd3e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:00.896389 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.896358    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-token" pod "45a767a0-a88a-4a84-b0e1-c918badbd3e9" (UID: "45a767a0-a88a-4a84-b0e1-c918badbd3e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.896528 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.896505    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-envoy" pod "45a767a0-a88a-4a84-b0e1-c918badbd3e9" (UID: "45a767a0-a88a-4a84-b0e1-c918badbd3e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.897060 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.897030    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-podinfo" pod "45a767a0-a88a-4a84-b0e1-c918badbd3e9" (UID: "45a767a0-a88a-4a84-b0e1-c918badbd3e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:00.897773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.897751    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a767a0-a88a-4a84-b0e1-c918badbd3e9-kube-api-access-mnt6m" pod "45a767a0-a88a-4a84-b0e1-c918badbd3e9" (UID: "45a767a0-a88a-4a84-b0e1-c918badbd3e9"). InnerVolumeSpecName "kube-api-access-mnt6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:00.915018 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.914977    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb" podUID="1df5b08d-07c1-4075-b8da-9545cabd30d9" containerName="payload-processing" containerID="cri-o://90c7e33b7e49aa29d31971eab8bcf9ef5f7fa665ab7ffe64c47ea26a4a761ce4" gracePeriod=25
Oct 30 13:39:00.986485 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.984297    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph" podUID="530acb52-18b3-4645-aad2-b5d56cc913f5" containerName="payload-pre-processing" containerID="cri-o://95a22e88c1413d1e15322e90a48b10ac13db6ce98228084e840228cecf33da0c" gracePeriod=25
Oct 30 13:39:00.996018 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.995986    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-workload-certs\") pod \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") "
Oct 30 13:39:00.996148 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.996076    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-kube-api-access-l9vmh\" (UniqueName: \"kubernetes.io/projected/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-kube-api-access-l9vmh\") pod \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") "
Oct 30 13:39:00.996189 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.996160    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-data\") pod \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") "
Oct 30 13:39:00.996253 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.996238    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-token\" (UniqueName: \"kubernetes.io/projected/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-token\") pod \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") "
Oct 30 13:39:00.997785 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.997705    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-workload-certs" pod "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" (UID: "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.997902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.997827    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-data" pod "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" (UID: "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:00.998437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.998323    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-credential-socket\") pod \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") "
Oct 30 13:39:00.999404 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.998605    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-workload-socket\") pod \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") "
Oct 30 13:39:00.999404 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.998771    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-podinfo\") pod \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") "
Oct 30 13:39:00.999404 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.998814    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istiod-ca-cert\") pod \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") "
Oct 30 13:39:00.999404 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:00.998847    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-envoy\") pod \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\" (UID: \"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d\") "
Oct 30 13:39:01.000321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.000021    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-credential-socket" pod "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" (UID: "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:01.000321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.000037    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.000321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.000077    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.000321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.000093    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.000321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.000108    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.000321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.000123    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.000321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.000136    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.000321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.000151    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45a767a0-a88a-4a84-b0e1-c918badbd3e9-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.000321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.000165    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnt6m\" (UniqueName: \"kubernetes.io/projected/45a767a0-a88a-4a84-b0e1-c918badbd3e9-kube-api-access-mnt6m\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.001020 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.000558    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istiod-ca-cert" pod "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" (UID: "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:01.001020 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.000682    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-workload-socket" pod "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" (UID: "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:01.003080 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.003037    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-podinfo" pod "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" (UID: "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:01.003181 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.003149    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-envoy" pod "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" (UID: "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:01.003243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.003187    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-kube-api-access-l9vmh" pod "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" (UID: "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d"). InnerVolumeSpecName "kube-api-access-l9vmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:01.005193 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.005155    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-token" pod "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" (UID: "f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:01.102136 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.102089    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.102136 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.102121    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.102136 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.102132    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.102136 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.102143    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.102495 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.102153    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.102495 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.102163    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.102495 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.102171    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9vmh\" (UniqueName: \"kubernetes.io/projected/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d-kube-api-access-l9vmh\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.318759 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.318723    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"
Oct 30 13:39:01.407682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.407557    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1df5b08d-07c1-4075-b8da-9545cabd30d9-kube-api-access-hkxm8\" (UniqueName: \"kubernetes.io/projected/1df5b08d-07c1-4075-b8da-9545cabd30d9-kube-api-access-hkxm8\") pod \"1df5b08d-07c1-4075-b8da-9545cabd30d9\" (UID: \"1df5b08d-07c1-4075-b8da-9545cabd30d9\") "
Oct 30 13:39:01.407897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.407874    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1df5b08d-07c1-4075-b8da-9545cabd30d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df5b08d-07c1-4075-b8da-9545cabd30d9-plugins-config-volume\") pod \"1df5b08d-07c1-4075-b8da-9545cabd30d9\" (UID: \"1df5b08d-07c1-4075-b8da-9545cabd30d9\") "
Oct 30 13:39:01.409356 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.409327    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df5b08d-07c1-4075-b8da-9545cabd30d9-plugins-config-volume" pod "1df5b08d-07c1-4075-b8da-9545cabd30d9" (UID: "1df5b08d-07c1-4075-b8da-9545cabd30d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:01.412021 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.411988    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df5b08d-07c1-4075-b8da-9545cabd30d9-kube-api-access-hkxm8" pod "1df5b08d-07c1-4075-b8da-9545cabd30d9" (UID: "1df5b08d-07c1-4075-b8da-9545cabd30d9"). InnerVolumeSpecName "kube-api-access-hkxm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:01.421943 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.421918    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"
Oct 30 13:39:01.509964 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.509926    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/530acb52-18b3-4645-aad2-b5d56cc913f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530acb52-18b3-4645-aad2-b5d56cc913f5-plugins-config-volume\") pod \"530acb52-18b3-4645-aad2-b5d56cc913f5\" (UID: \"530acb52-18b3-4645-aad2-b5d56cc913f5\") "
Oct 30 13:39:01.510216 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.510197    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/530acb52-18b3-4645-aad2-b5d56cc913f5-kube-api-access-8d57v\" (UniqueName: \"kubernetes.io/projected/530acb52-18b3-4645-aad2-b5d56cc913f5-kube-api-access-8d57v\") pod \"530acb52-18b3-4645-aad2-b5d56cc913f5\" (UID: \"530acb52-18b3-4645-aad2-b5d56cc913f5\") "
Oct 30 13:39:01.510842 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.510822    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1df5b08d-07c1-4075-b8da-9545cabd30d9-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.510961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.510847    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkxm8\" (UniqueName: \"kubernetes.io/projected/1df5b08d-07c1-4075-b8da-9545cabd30d9-kube-api-access-hkxm8\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.511605 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.511577    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/530acb52-18b3-4645-aad2-b5d56cc913f5-plugins-config-volume" pod "530acb52-18b3-4645-aad2-b5d56cc913f5" (UID: "530acb52-18b3-4645-aad2-b5d56cc913f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:01.515432 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.515405    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530acb52-18b3-4645-aad2-b5d56cc913f5-kube-api-access-8d57v" pod "530acb52-18b3-4645-aad2-b5d56cc913f5" (UID: "530acb52-18b3-4645-aad2-b5d56cc913f5"). InnerVolumeSpecName "kube-api-access-8d57v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:01.556892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.556851    5893 scope.go:122] "RemoveContainer" containerID="95a22e88c1413d1e15322e90a48b10ac13db6ce98228084e840228cecf33da0c"
Oct 30 13:39:01.580638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.580576    5893 scope.go:122] "RemoveContainer" containerID="90c7e33b7e49aa29d31971eab8bcf9ef5f7fa665ab7ffe64c47ea26a4a761ce4"
Oct 30 13:39:01.598527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.598498    5893 scope.go:122] "RemoveContainer" containerID="5da8786df647cbdbd6f953bc7ac7d16aa57775703b07ada147a2dabeb0c94961"
Oct 30 13:39:01.612915 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.612858    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d57v\" (UniqueName: \"kubernetes.io/projected/530acb52-18b3-4645-aad2-b5d56cc913f5-kube-api-access-8d57v\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.612915 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.612894    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/530acb52-18b3-4645-aad2-b5d56cc913f5-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:01.796840 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.796811    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"
Oct 30 13:39:01.797600 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.797579    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"
Oct 30 13:39:01.797719 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.797620    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb" event={"ID":"1df5b08d-07c1-4075-b8da-9545cabd30d9","Type":"ContainerDied","Data":"90c7e33b7e49aa29d31971eab8bcf9ef5f7fa665ab7ffe64c47ea26a4a761ce4"}
Oct 30 13:39:01.797719 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.797674    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb" event={"ID":"1df5b08d-07c1-4075-b8da-9545cabd30d9","Type":"ContainerDied","Data":"d46b31a45f6b604e64e9c4caa112351fe64f016144b4952882e9f3cc13302333"}
Oct 30 13:39:01.797719 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.797697    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws" event={"ID":"f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d","Type":"ContainerDied","Data":"db8c62f48d0612875e5b103ecf3e7a411a8ace6b01a5cef13cab996d0e3f9807"}
Oct 30 13:39:01.797873 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.797728    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph" event={"ID":"530acb52-18b3-4645-aad2-b5d56cc913f5","Type":"ContainerDied","Data":"95a22e88c1413d1e15322e90a48b10ac13db6ce98228084e840228cecf33da0c"}
Oct 30 13:39:01.797873 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.797785    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph" event={"ID":"530acb52-18b3-4645-aad2-b5d56cc913f5","Type":"ContainerDied","Data":"b2583444b346b3615b3a2fa1a343ffa6a1d067d9afa76d4db684c3b7538ef779"}
Oct 30 13:39:01.797873 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.797806    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"
Oct 30 13:39:01.798012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.797878    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"
Oct 30 13:39:01.877832 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.877798    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"]
Oct 30 13:39:01.885987 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.885358    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-96ab73-d779d96b4-9v8bb"]
Oct 30 13:39:01.952008 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.951916    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"]
Oct 30 13:39:01.956949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:01.956916    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-0e874e-openshift-default-594f895d85-hx4ws"]
Oct 30 13:39:02.007031 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:02.006997    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"]
Oct 30 13:39:02.015084 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:02.015057    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-96ab73-56bf464d67-cjgph"]
Oct 30 13:39:02.047559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:02.047531    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"]
Oct 30 13:39:02.052852 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:02.052823    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3b47e7-openshift-default-64f995d48c-65l6w"]
Oct 30 13:39:02.536828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:02.536787    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1df5b08d-07c1-4075-b8da-9545cabd30d9" path="/var/lib/kubelet/pods/1df5b08d-07c1-4075-b8da-9545cabd30d9/volumes"
Oct 30 13:39:02.537377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:02.537311    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" path="/var/lib/kubelet/pods/45a767a0-a88a-4a84-b0e1-c918badbd3e9/volumes"
Oct 30 13:39:02.537706 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:02.537688    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="530acb52-18b3-4645-aad2-b5d56cc913f5" path="/var/lib/kubelet/pods/530acb52-18b3-4645-aad2-b5d56cc913f5/volumes"
Oct 30 13:39:02.538446 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:02.538417    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e678e9e-ec06-4b9e-b1f6-cebb61e59571" path="/var/lib/kubelet/pods/5e678e9e-ec06-4b9e-b1f6-cebb61e59571/volumes"
Oct 30 13:39:02.538872 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:02.538855    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="895b7a55-b271-4b61-a3c5-7411e8b9a9b0" path="/var/lib/kubelet/pods/895b7a55-b271-4b61-a3c5-7411e8b9a9b0/volumes"
Oct 30 13:39:02.539333 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:02.539318    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" path="/var/lib/kubelet/pods/f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d/volumes"
Oct 30 13:39:03.814765 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:03.814730    5893 generic.go:356] "Generic (PLEG): container finished" podID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerID="810c3a6e5a613dee41706a5a570f87dc2b5e4c4a47b683e1575b29506ff32813" exitCode=0
Oct 30 13:39:03.815286 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:03.814802    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" event={"ID":"6e02b51d-4bfe-4805-b52e-d65419291f1e","Type":"ContainerDied","Data":"810c3a6e5a613dee41706a5a570f87dc2b5e4c4a47b683e1575b29506ff32813"}
Oct 30 13:39:03.815369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:03.815329    5893 scope.go:122] "RemoveContainer" containerID="810c3a6e5a613dee41706a5a570f87dc2b5e4c4a47b683e1575b29506ff32813"
Oct 30 13:39:03.817078 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:03.817051    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:39:03.817150 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:03.817124    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:39:03.817191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:03.817162    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:39:04.216026 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.215822    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:39:04.357582 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.357509    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:39:04.362408 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.362383    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:39:04.388738 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.388709    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-istio-envoy\") pod \"728d41af-901f-4def-a2c1-aee79df80e1e\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") "
Oct 30 13:39:04.388927 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.388782    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/728d41af-901f-4def-a2c1-aee79df80e1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728d41af-901f-4def-a2c1-aee79df80e1e-istio-podinfo\") pod \"728d41af-901f-4def-a2c1-aee79df80e1e\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") "
Oct 30 13:39:04.388927 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.388816    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-credential-socket\") pod \"728d41af-901f-4def-a2c1-aee79df80e1e\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") "
Oct 30 13:39:04.388927 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.388867    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-istio-data\") pod \"728d41af-901f-4def-a2c1-aee79df80e1e\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") "
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.388968    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-workload-socket\") pod \"728d41af-901f-4def-a2c1-aee79df80e1e\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") "
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.389159    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-credential-socket" pod "728d41af-901f-4def-a2c1-aee79df80e1e" (UID: "728d41af-901f-4def-a2c1-aee79df80e1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.389207    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/728d41af-901f-4def-a2c1-aee79df80e1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728d41af-901f-4def-a2c1-aee79df80e1e-istiod-ca-cert\") pod \"728d41af-901f-4def-a2c1-aee79df80e1e\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") "
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.389251    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-istio-data" pod "728d41af-901f-4def-a2c1-aee79df80e1e" (UID: "728d41af-901f-4def-a2c1-aee79df80e1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.389319    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-workload-socket" pod "728d41af-901f-4def-a2c1-aee79df80e1e" (UID: "728d41af-901f-4def-a2c1-aee79df80e1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.389363    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-workload-certs\") pod \"728d41af-901f-4def-a2c1-aee79df80e1e\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") "
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.389416    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728d41af-901f-4def-a2c1-aee79df80e1e-kube-api-access-54tms\" (UniqueName: \"kubernetes.io/projected/728d41af-901f-4def-a2c1-aee79df80e1e-kube-api-access-54tms\") pod \"728d41af-901f-4def-a2c1-aee79df80e1e\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") "
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.389522    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/728d41af-901f-4def-a2c1-aee79df80e1e-istio-token\" (UniqueName: \"kubernetes.io/projected/728d41af-901f-4def-a2c1-aee79df80e1e-istio-token\") pod \"728d41af-901f-4def-a2c1-aee79df80e1e\" (UID: \"728d41af-901f-4def-a2c1-aee79df80e1e\") "
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.389543    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-workload-certs" pod "728d41af-901f-4def-a2c1-aee79df80e1e" (UID: "728d41af-901f-4def-a2c1-aee79df80e1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.389566    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/728d41af-901f-4def-a2c1-aee79df80e1e-istiod-ca-cert" pod "728d41af-901f-4def-a2c1-aee79df80e1e" (UID: "728d41af-901f-4def-a2c1-aee79df80e1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.390456    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.390478    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.390494    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.390508    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.390547 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.390521    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/728d41af-901f-4def-a2c1-aee79df80e1e-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.391709 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.391681    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-istio-envoy" pod "728d41af-901f-4def-a2c1-aee79df80e1e" (UID: "728d41af-901f-4def-a2c1-aee79df80e1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.392738 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.392716    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728d41af-901f-4def-a2c1-aee79df80e1e-kube-api-access-54tms" pod "728d41af-901f-4def-a2c1-aee79df80e1e" (UID: "728d41af-901f-4def-a2c1-aee79df80e1e"). InnerVolumeSpecName "kube-api-access-54tms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:04.392826 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.392804    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/728d41af-901f-4def-a2c1-aee79df80e1e-istio-token" pod "728d41af-901f-4def-a2c1-aee79df80e1e" (UID: "728d41af-901f-4def-a2c1-aee79df80e1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:04.393635 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.393612    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/728d41af-901f-4def-a2c1-aee79df80e1e-istio-podinfo" pod "728d41af-901f-4def-a2c1-aee79df80e1e" (UID: "728d41af-901f-4def-a2c1-aee79df80e1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:04.491600 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.491566    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-token\" (UniqueName: \"kubernetes.io/projected/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-token\") pod \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") "
Oct 30 13:39:04.491817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.491620    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istiod-ca-cert\") pod \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") "
Oct 30 13:39:04.491817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.491656    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-workload-certs\") pod \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") "
Oct 30 13:39:04.491817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.491692    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-workload-certs\") pod \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") "
Oct 30 13:39:04.491817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.491809    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-workload-socket\") pod \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") "
Oct 30 13:39:04.492076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.491872    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-kube-api-access-bn2f6\" (UniqueName: \"kubernetes.io/projected/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-kube-api-access-bn2f6\") pod \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") "
Oct 30 13:39:04.492076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.491901    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-podinfo\") pod \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") "
Oct 30 13:39:04.492076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.491965    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-envoy\") pod \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") "
Oct 30 13:39:04.492076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.491995    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-envoy\") pod \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") "
Oct 30 13:39:04.492076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.492053    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-podinfo\") pod \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") "
Oct 30 13:39:04.492076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.492073    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-workload-socket\") pod \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") "
Oct 30 13:39:04.492435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.492090    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-data\") pod \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") "
Oct 30 13:39:04.492435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.492112    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istiod-ca-cert\") pod \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") "
Oct 30 13:39:04.492435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.492134    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-kube-api-access-z68wn\" (UniqueName: \"kubernetes.io/projected/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-kube-api-access-z68wn\") pod \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") "
Oct 30 13:39:04.492435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.492156    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-workload-certs" pod "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" (UID: "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.492435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.492191    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-credential-socket\") pod \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") "
Oct 30 13:39:04.492435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.492263    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-credential-socket\") pod \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") "
Oct 30 13:39:04.492435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.492298    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-data\") pod \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\" (UID: \"8b1530d7-d75e-4d05-9fda-46f3a07b60c1\") "
Oct 30 13:39:04.492435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.492400    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-token\" (UniqueName: \"kubernetes.io/projected/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-token\") pod \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\" (UID: \"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa\") "
Oct 30 13:39:04.492435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.492427    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-workload-certs" pod "8b1530d7-d75e-4d05-9fda-46f3a07b60c1" (UID: "8b1530d7-d75e-4d05-9fda-46f3a07b60c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.492932 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.492616    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-workload-socket" pod "8b1530d7-d75e-4d05-9fda-46f3a07b60c1" (UID: "8b1530d7-d75e-4d05-9fda-46f3a07b60c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.493148 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.493121    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/728d41af-901f-4def-a2c1-aee79df80e1e-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.493245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.493155    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.493245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.493170    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/728d41af-901f-4def-a2c1-aee79df80e1e-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.493245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.493184    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/728d41af-901f-4def-a2c1-aee79df80e1e-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.493245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.493197    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.493245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.493210    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.493245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.493223    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54tms\" (UniqueName: \"kubernetes.io/projected/728d41af-901f-4def-a2c1-aee79df80e1e-kube-api-access-54tms\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.493606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.493558    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istiod-ca-cert" pod "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" (UID: "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:04.493899 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.493877    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-workload-socket" pod "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" (UID: "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.494083 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.494051    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-token" pod "8b1530d7-d75e-4d05-9fda-46f3a07b60c1" (UID: "8b1530d7-d75e-4d05-9fda-46f3a07b60c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:04.494550 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.494526    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-credential-socket" pod "8b1530d7-d75e-4d05-9fda-46f3a07b60c1" (UID: "8b1530d7-d75e-4d05-9fda-46f3a07b60c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.494825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.494803    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-credential-socket" pod "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" (UID: "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.495033 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.495005    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-data" pod "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" (UID: "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.495144 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.495053    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-token" pod "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" (UID: "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:04.495559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.495534    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istiod-ca-cert" pod "8b1530d7-d75e-4d05-9fda-46f3a07b60c1" (UID: "8b1530d7-d75e-4d05-9fda-46f3a07b60c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:04.496198 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.496172    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-envoy" pod "8b1530d7-d75e-4d05-9fda-46f3a07b60c1" (UID: "8b1530d7-d75e-4d05-9fda-46f3a07b60c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.496449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.496425    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-data" pod "8b1530d7-d75e-4d05-9fda-46f3a07b60c1" (UID: "8b1530d7-d75e-4d05-9fda-46f3a07b60c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.496951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.496913    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-envoy" pod "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" (UID: "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:04.497379 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.497346    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-kube-api-access-bn2f6" pod "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" (UID: "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa"). InnerVolumeSpecName "kube-api-access-bn2f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:04.497569 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.497550    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-podinfo" pod "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" (UID: "c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:04.497665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.497649    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-kube-api-access-z68wn" pod "8b1530d7-d75e-4d05-9fda-46f3a07b60c1" (UID: "8b1530d7-d75e-4d05-9fda-46f3a07b60c1"). InnerVolumeSpecName "kube-api-access-z68wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:04.498389 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.498370    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-podinfo" pod "8b1530d7-d75e-4d05-9fda-46f3a07b60c1" (UID: "8b1530d7-d75e-4d05-9fda-46f3a07b60c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:04.595178 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595146    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.595178 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595175    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn2f6\" (UniqueName: \"kubernetes.io/projected/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-kube-api-access-bn2f6\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.595178 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595184    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.595458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595195    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.595458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595203    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.595458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595210    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.595458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595218    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.595458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595226    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.595458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595234    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z68wn\" (UniqueName: \"kubernetes.io/projected/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-kube-api-access-z68wn\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.595458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595242    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.595458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595250    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.595458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595258    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.595458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595267    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.595458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595295    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b1530d7-d75e-4d05-9fda-46f3a07b60c1-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.595458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.595302    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:04.822566 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.822533    5893 generic.go:356] "Generic (PLEG): container finished" podID="c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" containerID="b411062db69331b015ac4bb313a3c748070c4004fafd994ff950d237006ee4c2" exitCode=0
Oct 30 13:39:04.823068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.822583    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"
Oct 30 13:39:04.823068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.822619    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp" event={"ID":"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa","Type":"ContainerDied","Data":"b411062db69331b015ac4bb313a3c748070c4004fafd994ff950d237006ee4c2"}
Oct 30 13:39:04.823068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.822675    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp" event={"ID":"c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa","Type":"ContainerDied","Data":"dc982469c0af36a148cf29aebdcdab78aaff8e4ca4272bc922d4ce1565ca3971"}
Oct 30 13:39:04.823068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.822703    5893 scope.go:122] "RemoveContainer" containerID="b411062db69331b015ac4bb313a3c748070c4004fafd994ff950d237006ee4c2"
Oct 30 13:39:04.825241 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.825209    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" event={"ID":"6e02b51d-4bfe-4805-b52e-d65419291f1e","Type":"ContainerStarted","Data":"53dfc999504f7a52c08ad43badd0abdc57eca93341cd81befea412dfa4b951de"}
Oct 30 13:39:04.827127 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.827105    5893 generic.go:356] "Generic (PLEG): container finished" podID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerID="43ec74448547cda39d22fbe5e3d1b0b1bc4e335bf70251a9bcb78a7f7170cf96" exitCode=0
Oct 30 13:39:04.827227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.827154    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"
Oct 30 13:39:04.827227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.827189    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" event={"ID":"8b1530d7-d75e-4d05-9fda-46f3a07b60c1","Type":"ContainerDied","Data":"43ec74448547cda39d22fbe5e3d1b0b1bc4e335bf70251a9bcb78a7f7170cf96"}
Oct 30 13:39:04.827355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.827234    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj" event={"ID":"8b1530d7-d75e-4d05-9fda-46f3a07b60c1","Type":"ContainerDied","Data":"470f973c32bff4c74b3a652cb38cf741e7decc47dcd61223c9b0a16022c41e24"}
Oct 30 13:39:04.828700 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.828681    5893 generic.go:356] "Generic (PLEG): container finished" podID="728d41af-901f-4def-a2c1-aee79df80e1e" containerID="30ad6e481111954c9d55bc109f6d7048332b46fdfecd5ae5bb7fd9ca76069fe8" exitCode=0
Oct 30 13:39:04.828806 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.828718    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"
Oct 30 13:39:04.828806 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.828740    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj" event={"ID":"728d41af-901f-4def-a2c1-aee79df80e1e","Type":"ContainerDied","Data":"30ad6e481111954c9d55bc109f6d7048332b46fdfecd5ae5bb7fd9ca76069fe8"}
Oct 30 13:39:04.828806 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.828790    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj" event={"ID":"728d41af-901f-4def-a2c1-aee79df80e1e","Type":"ContainerDied","Data":"d28408242921370e865c5f0682f09a7267a92fa86ca30e8e1aad8163fd4ef5b1"}
Oct 30 13:39:04.835744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.835730    5893 scope.go:122] "RemoveContainer" containerID="b411062db69331b015ac4bb313a3c748070c4004fafd994ff950d237006ee4c2"
Oct 30 13:39:04.836045 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:04.836021    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b411062db69331b015ac4bb313a3c748070c4004fafd994ff950d237006ee4c2\": container with ID starting with b411062db69331b015ac4bb313a3c748070c4004fafd994ff950d237006ee4c2 not found: ID does not exist" containerID="b411062db69331b015ac4bb313a3c748070c4004fafd994ff950d237006ee4c2"
Oct 30 13:39:04.836149 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.836051    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b411062db69331b015ac4bb313a3c748070c4004fafd994ff950d237006ee4c2"} err="failed to get container status \"b411062db69331b015ac4bb313a3c748070c4004fafd994ff950d237006ee4c2\": rpc error: code = NotFound desc = could not find container \"b411062db69331b015ac4bb313a3c748070c4004fafd994ff950d237006ee4c2\": container with ID starting with b411062db69331b015ac4bb313a3c748070c4004fafd994ff950d237006ee4c2 not found: ID does not exist"
Oct 30 13:39:04.836149 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.836080    5893 scope.go:122] "RemoveContainer" containerID="43ec74448547cda39d22fbe5e3d1b0b1bc4e335bf70251a9bcb78a7f7170cf96"
Oct 30 13:39:04.871308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.870614    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:39:04.874599 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.874090    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 30 13:39:04.874599 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.874173    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 30 13:39:04.877166 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.877135    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"]
Oct 30 13:39:04.883004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.882988    5893 scope.go:122] "RemoveContainer" containerID="6760e5c07c8e77766167a484dccf47838318d578fd146c9395330692f1de6431"
Oct 30 13:39:04.890425 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.890398    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-8x9rj"]
Oct 30 13:39:04.897954 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.897936    5893 scope.go:122] "RemoveContainer" containerID="43ec74448547cda39d22fbe5e3d1b0b1bc4e335bf70251a9bcb78a7f7170cf96"
Oct 30 13:39:04.898218 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:04.898199    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ec74448547cda39d22fbe5e3d1b0b1bc4e335bf70251a9bcb78a7f7170cf96\": container with ID starting with 43ec74448547cda39d22fbe5e3d1b0b1bc4e335bf70251a9bcb78a7f7170cf96 not found: ID does not exist" containerID="43ec74448547cda39d22fbe5e3d1b0b1bc4e335bf70251a9bcb78a7f7170cf96"
Oct 30 13:39:04.898315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.898228    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ec74448547cda39d22fbe5e3d1b0b1bc4e335bf70251a9bcb78a7f7170cf96"} err="failed to get container status \"43ec74448547cda39d22fbe5e3d1b0b1bc4e335bf70251a9bcb78a7f7170cf96\": rpc error: code = NotFound desc = could not find container \"43ec74448547cda39d22fbe5e3d1b0b1bc4e335bf70251a9bcb78a7f7170cf96\": container with ID starting with 43ec74448547cda39d22fbe5e3d1b0b1bc4e335bf70251a9bcb78a7f7170cf96 not found: ID does not exist"
Oct 30 13:39:04.898315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.898246    5893 scope.go:122] "RemoveContainer" containerID="6760e5c07c8e77766167a484dccf47838318d578fd146c9395330692f1de6431"
Oct 30 13:39:04.898519 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:04.898503    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6760e5c07c8e77766167a484dccf47838318d578fd146c9395330692f1de6431\": container with ID starting with 6760e5c07c8e77766167a484dccf47838318d578fd146c9395330692f1de6431 not found: ID does not exist" containerID="6760e5c07c8e77766167a484dccf47838318d578fd146c9395330692f1de6431"
Oct 30 13:39:04.898561 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.898523    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6760e5c07c8e77766167a484dccf47838318d578fd146c9395330692f1de6431"} err="failed to get container status \"6760e5c07c8e77766167a484dccf47838318d578fd146c9395330692f1de6431\": rpc error: code = NotFound desc = could not find container \"6760e5c07c8e77766167a484dccf47838318d578fd146c9395330692f1de6431\": container with ID starting with 6760e5c07c8e77766167a484dccf47838318d578fd146c9395330692f1de6431 not found: ID does not exist"
Oct 30 13:39:04.898561 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.898538    5893 scope.go:122] "RemoveContainer" containerID="30ad6e481111954c9d55bc109f6d7048332b46fdfecd5ae5bb7fd9ca76069fe8"
Oct 30 13:39:04.909174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.909157    5893 scope.go:122] "RemoveContainer" containerID="30ad6e481111954c9d55bc109f6d7048332b46fdfecd5ae5bb7fd9ca76069fe8"
Oct 30 13:39:04.909466 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:04.909444    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30ad6e481111954c9d55bc109f6d7048332b46fdfecd5ae5bb7fd9ca76069fe8\": container with ID starting with 30ad6e481111954c9d55bc109f6d7048332b46fdfecd5ae5bb7fd9ca76069fe8 not found: ID does not exist" containerID="30ad6e481111954c9d55bc109f6d7048332b46fdfecd5ae5bb7fd9ca76069fe8"
Oct 30 13:39:04.909529 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:04.909476    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30ad6e481111954c9d55bc109f6d7048332b46fdfecd5ae5bb7fd9ca76069fe8"} err="failed to get container status \"30ad6e481111954c9d55bc109f6d7048332b46fdfecd5ae5bb7fd9ca76069fe8\": rpc error: code = NotFound desc = could not find container \"30ad6e481111954c9d55bc109f6d7048332b46fdfecd5ae5bb7fd9ca76069fe8\": container with ID starting with 30ad6e481111954c9d55bc109f6d7048332b46fdfecd5ae5bb7fd9ca76069fe8 not found: ID does not exist"
Oct 30 13:39:05.011509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.011468    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"]
Oct 30 13:39:05.014412 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.014385    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-lnsxj"]
Oct 30 13:39:05.101347 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.100331    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"]
Oct 30 13:39:05.104372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.104324    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1c8130-openshift-default-76f8764dd5-p8lqp"]
Oct 30 13:39:05.520593 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.520570    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:39:05.615245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.615210    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-podinfo\") pod \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") "
Oct 30 13:39:05.615449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.615254    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-kube-api-access-xqsbh\" (UniqueName: \"kubernetes.io/projected/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-kube-api-access-xqsbh\") pod \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") "
Oct 30 13:39:05.615449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.615323    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-workload-socket\") pod \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") "
Oct 30 13:39:05.615449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.615426    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-data\") pod \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") "
Oct 30 13:39:05.615650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.615488    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-workload-certs\") pod \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") "
Oct 30 13:39:05.615650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.615514    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-credential-socket\") pod \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") "
Oct 30 13:39:05.615650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.615603    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-token\" (UniqueName: \"kubernetes.io/projected/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-token\") pod \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") "
Oct 30 13:39:05.615650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.615640    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istiod-ca-cert\") pod \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") "
Oct 30 13:39:05.615880 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.615716    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-envoy\") pod \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\" (UID: \"8525f4ea-caea-4321-8c2e-4c6fa4fe06da\") "
Oct 30 13:39:05.615880 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.615741    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-workload-socket" pod "8525f4ea-caea-4321-8c2e-4c6fa4fe06da" (UID: "8525f4ea-caea-4321-8c2e-4c6fa4fe06da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.615989 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.615903    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-credential-socket" pod "8525f4ea-caea-4321-8c2e-4c6fa4fe06da" (UID: "8525f4ea-caea-4321-8c2e-4c6fa4fe06da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.616035 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.615996    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-data" pod "8525f4ea-caea-4321-8c2e-4c6fa4fe06da" (UID: "8525f4ea-caea-4321-8c2e-4c6fa4fe06da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.616035 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.616009    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-workload-certs" pod "8525f4ea-caea-4321-8c2e-4c6fa4fe06da" (UID: "8525f4ea-caea-4321-8c2e-4c6fa4fe06da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.616222 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.616196    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istiod-ca-cert" pod "8525f4ea-caea-4321-8c2e-4c6fa4fe06da" (UID: "8525f4ea-caea-4321-8c2e-4c6fa4fe06da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:05.616619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.616537    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.616619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.616567    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.616619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.616587    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.616619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.616605    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.616952 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.616675    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.618571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.618548    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-kube-api-access-xqsbh" pod "8525f4ea-caea-4321-8c2e-4c6fa4fe06da" (UID: "8525f4ea-caea-4321-8c2e-4c6fa4fe06da"). InnerVolumeSpecName "kube-api-access-xqsbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:05.618809 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.618786    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-token" pod "8525f4ea-caea-4321-8c2e-4c6fa4fe06da" (UID: "8525f4ea-caea-4321-8c2e-4c6fa4fe06da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:05.618949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.618906    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-podinfo" pod "8525f4ea-caea-4321-8c2e-4c6fa4fe06da" (UID: "8525f4ea-caea-4321-8c2e-4c6fa4fe06da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:05.618949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.618925    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-envoy" pod "8525f4ea-caea-4321-8c2e-4c6fa4fe06da" (UID: "8525f4ea-caea-4321-8c2e-4c6fa4fe06da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:05.718517 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.718488    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.718517 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.718516    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqsbh\" (UniqueName: \"kubernetes.io/projected/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-kube-api-access-xqsbh\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.718737 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.718527    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.718737 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.718536    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8525f4ea-caea-4321-8c2e-4c6fa4fe06da-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:05.838561 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.838525    5893 generic.go:356] "Generic (PLEG): container finished" podID="8525f4ea-caea-4321-8c2e-4c6fa4fe06da" containerID="610b534d7f749ae4444f821a6513c40fdbb0a774cd3e09093c088aac338228f7" exitCode=0
Oct 30 13:39:05.838995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.838567    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"
Oct 30 13:39:05.838995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.838601    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln" event={"ID":"8525f4ea-caea-4321-8c2e-4c6fa4fe06da","Type":"ContainerDied","Data":"610b534d7f749ae4444f821a6513c40fdbb0a774cd3e09093c088aac338228f7"}
Oct 30 13:39:05.838995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.838663    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln" event={"ID":"8525f4ea-caea-4321-8c2e-4c6fa4fe06da","Type":"ContainerDied","Data":"75f1828568104c242b76625abbf5742aee1b1d7aa1ff6b42f55b8c9c3384190f"}
Oct 30 13:39:05.838995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.838697    5893 scope.go:122] "RemoveContainer" containerID="610b534d7f749ae4444f821a6513c40fdbb0a774cd3e09093c088aac338228f7"
Oct 30 13:39:05.880327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.878861    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 30 13:39:05.880327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.878920    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 30 13:39:05.880327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.879798    5893 scope.go:122] "RemoveContainer" containerID="610b534d7f749ae4444f821a6513c40fdbb0a774cd3e09093c088aac338228f7"
Oct 30 13:39:05.880327 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:05.880165    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"610b534d7f749ae4444f821a6513c40fdbb0a774cd3e09093c088aac338228f7\": container with ID starting with 610b534d7f749ae4444f821a6513c40fdbb0a774cd3e09093c088aac338228f7 not found: ID does not exist" containerID="610b534d7f749ae4444f821a6513c40fdbb0a774cd3e09093c088aac338228f7"
Oct 30 13:39:05.880327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.880204    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"610b534d7f749ae4444f821a6513c40fdbb0a774cd3e09093c088aac338228f7"} err="failed to get container status \"610b534d7f749ae4444f821a6513c40fdbb0a774cd3e09093c088aac338228f7\": rpc error: code = NotFound desc = could not find container \"610b534d7f749ae4444f821a6513c40fdbb0a774cd3e09093c088aac338228f7\": container with ID starting with 610b534d7f749ae4444f821a6513c40fdbb0a774cd3e09093c088aac338228f7 not found: ID does not exist"
Oct 30 13:39:05.919372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.919340    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"]
Oct 30 13:39:05.929228 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:05.929197    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-0b2c92-openshift-default-55b4bd6c47-6qdln"]
Oct 30 13:39:06.532595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:06.532564    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="728d41af-901f-4def-a2c1-aee79df80e1e" path="/var/lib/kubelet/pods/728d41af-901f-4def-a2c1-aee79df80e1e/volumes"
Oct 30 13:39:06.533087 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:06.533072    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8525f4ea-caea-4321-8c2e-4c6fa4fe06da" path="/var/lib/kubelet/pods/8525f4ea-caea-4321-8c2e-4c6fa4fe06da/volumes"
Oct 30 13:39:06.533588 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:06.533574    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" path="/var/lib/kubelet/pods/8b1530d7-d75e-4d05-9fda-46f3a07b60c1/volumes"
Oct 30 13:39:06.534507 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:06.534477    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" path="/var/lib/kubelet/pods/c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa/volumes"
Oct 30 13:39:06.871353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:06.870460    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 30 13:39:06.871353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:06.870524    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 30 13:39:07.879635 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:07.879594    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 30 13:39:07.880181 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:07.879659    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 30 13:39:08.837461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:08.837410    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:39:08.883813 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:08.883777    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 30 13:39:08.926702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:08.926647    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 30 13:39:09.868893 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:09.868849    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused" start-of-body=
Oct 30 13:39:09.869103 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:09.868965    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.202:15021/healthz/ready\": dial tcp 10.134.0.202:15021: connect: connection refused"
Oct 30 13:39:10.873663 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:10.873632    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:39:10.874992 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:10.874965    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:39:11.230474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:11.230417    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"]
Oct 30 13:39:11.231085 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:11.231031    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql" podUID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerName="istio-proxy" containerID="cri-o://be190cf826b3356ece2e753fe3e0efc84bfd8139859a57b0765d2ceb1b78ed76" gracePeriod=30
Oct 30 13:39:11.252474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:11.252190    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"]
Oct 30 13:39:11.258199 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:11.255893    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj" podUID="805ec71c-0b0c-48a9-a154-7fbf568f73ba" containerName="istio-proxy" containerID="cri-o://cc5765085692b031c63afac12f8909598aab3f7a7fc7486109de03abad941fcd" gracePeriod=30
Oct 30 13:39:12.836186 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:12.836153    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"]
Oct 30 13:39:12.880997 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:12.880957    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"]
Oct 30 13:39:13.534370 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:13.534332    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"]
Oct 30 13:39:13.534623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:13.534577    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js" podUID="ddd8433a-82e7-43f7-a036-1c79a64230a1" containerName="payload-processing" containerID="cri-o://2ddd29bf0e636f56dd14a32ed6edce1aba855c0f977142dea71556fd134626f6" gracePeriod=30
Oct 30 13:39:13.596298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:13.595360    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"]
Oct 30 13:39:13.596298 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:13.595638    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj" podUID="21bd8ee2-cb78-4333-bccc-e1c7b34286c7" containerName="payload-pre-processing" containerID="cri-o://4e3f93b8d5d6cdceac21b5777955ad85c5a4377c93d5c4c72d25e1074e234db6" gracePeriod=30
Oct 30 13:39:13.862302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:13.862255    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"
Oct 30 13:39:13.957145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:13.948758    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd8433a-82e7-43f7-a036-1c79a64230a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd8433a-82e7-43f7-a036-1c79a64230a1-plugins-config-volume\") pod \"ddd8433a-82e7-43f7-a036-1c79a64230a1\" (UID: \"ddd8433a-82e7-43f7-a036-1c79a64230a1\") "
Oct 30 13:39:13.957145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:13.948827    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd8433a-82e7-43f7-a036-1c79a64230a1-kube-api-access-q8lkv\" (UniqueName: \"kubernetes.io/projected/ddd8433a-82e7-43f7-a036-1c79a64230a1-kube-api-access-q8lkv\") pod \"ddd8433a-82e7-43f7-a036-1c79a64230a1\" (UID: \"ddd8433a-82e7-43f7-a036-1c79a64230a1\") "
Oct 30 13:39:13.957145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:13.951171    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd8433a-82e7-43f7-a036-1c79a64230a1-plugins-config-volume" pod "ddd8433a-82e7-43f7-a036-1c79a64230a1" (UID: "ddd8433a-82e7-43f7-a036-1c79a64230a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:13.957145 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:13.953085    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd8433a-82e7-43f7-a036-1c79a64230a1-kube-api-access-q8lkv" pod "ddd8433a-82e7-43f7-a036-1c79a64230a1" (UID: "ddd8433a-82e7-43f7-a036-1c79a64230a1"). InnerVolumeSpecName "kube-api-access-q8lkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:13.990378 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:13.990041    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"
Oct 30 13:39:14.002991 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.000461    5893 generic.go:356] "Generic (PLEG): container finished" podID="ddd8433a-82e7-43f7-a036-1c79a64230a1" containerID="2ddd29bf0e636f56dd14a32ed6edce1aba855c0f977142dea71556fd134626f6" exitCode=0
Oct 30 13:39:14.002991 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.000576    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js" event={"ID":"ddd8433a-82e7-43f7-a036-1c79a64230a1","Type":"ContainerDied","Data":"2ddd29bf0e636f56dd14a32ed6edce1aba855c0f977142dea71556fd134626f6"}
Oct 30 13:39:14.002991 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.000618    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js" event={"ID":"ddd8433a-82e7-43f7-a036-1c79a64230a1","Type":"ContainerDied","Data":"fb56a5d7cf7f476a47a2891fd7f64c9e52ed2b29fca9aa9cb632a6a85028c19a"}
Oct 30 13:39:14.002991 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.000638    5893 scope.go:122] "RemoveContainer" containerID="2ddd29bf0e636f56dd14a32ed6edce1aba855c0f977142dea71556fd134626f6"
Oct 30 13:39:14.002991 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.000752    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"
Oct 30 13:39:14.007669 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.007637    5893 generic.go:356] "Generic (PLEG): container finished" podID="21bd8ee2-cb78-4333-bccc-e1c7b34286c7" containerID="4e3f93b8d5d6cdceac21b5777955ad85c5a4377c93d5c4c72d25e1074e234db6" exitCode=0
Oct 30 13:39:14.007802 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.007781    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj" event={"ID":"21bd8ee2-cb78-4333-bccc-e1c7b34286c7","Type":"ContainerDied","Data":"4e3f93b8d5d6cdceac21b5777955ad85c5a4377c93d5c4c72d25e1074e234db6"}
Oct 30 13:39:14.007866 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.007834    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj" event={"ID":"21bd8ee2-cb78-4333-bccc-e1c7b34286c7","Type":"ContainerDied","Data":"e7b09449f98bd61a9d3795a9506a92ef41fe92f83371764ab596b86c03c539ff"}
Oct 30 13:39:14.007938 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.007920    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"
Oct 30 13:39:14.055454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.053376    5893 scope.go:122] "RemoveContainer" containerID="2ddd29bf0e636f56dd14a32ed6edce1aba855c0f977142dea71556fd134626f6"
Oct 30 13:39:14.055454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.053932    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21bd8ee2-cb78-4333-bccc-e1c7b34286c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bd8ee2-cb78-4333-bccc-e1c7b34286c7-plugins-config-volume\") pod \"21bd8ee2-cb78-4333-bccc-e1c7b34286c7\" (UID: \"21bd8ee2-cb78-4333-bccc-e1c7b34286c7\") "
Oct 30 13:39:14.055454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.054135    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21bd8ee2-cb78-4333-bccc-e1c7b34286c7-kube-api-access-2j96p\" (UniqueName: \"kubernetes.io/projected/21bd8ee2-cb78-4333-bccc-e1c7b34286c7-kube-api-access-2j96p\") pod \"21bd8ee2-cb78-4333-bccc-e1c7b34286c7\" (UID: \"21bd8ee2-cb78-4333-bccc-e1c7b34286c7\") "
Oct 30 13:39:14.055454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.054727    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddd8433a-82e7-43f7-a036-1c79a64230a1-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:14.055454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.054750    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8lkv\" (UniqueName: \"kubernetes.io/projected/ddd8433a-82e7-43f7-a036-1c79a64230a1-kube-api-access-q8lkv\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:14.058266 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:14.058223    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ddd29bf0e636f56dd14a32ed6edce1aba855c0f977142dea71556fd134626f6\": container with ID starting with 2ddd29bf0e636f56dd14a32ed6edce1aba855c0f977142dea71556fd134626f6 not found: ID does not exist" containerID="2ddd29bf0e636f56dd14a32ed6edce1aba855c0f977142dea71556fd134626f6"
Oct 30 13:39:14.058266 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.058292    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ddd29bf0e636f56dd14a32ed6edce1aba855c0f977142dea71556fd134626f6"} err="failed to get container status \"2ddd29bf0e636f56dd14a32ed6edce1aba855c0f977142dea71556fd134626f6\": rpc error: code = NotFound desc = could not find container \"2ddd29bf0e636f56dd14a32ed6edce1aba855c0f977142dea71556fd134626f6\": container with ID starting with 2ddd29bf0e636f56dd14a32ed6edce1aba855c0f977142dea71556fd134626f6 not found: ID does not exist"
Oct 30 13:39:14.058540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.058320    5893 scope.go:122] "RemoveContainer" containerID="4e3f93b8d5d6cdceac21b5777955ad85c5a4377c93d5c4c72d25e1074e234db6"
Oct 30 13:39:14.059134 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.059102    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21bd8ee2-cb78-4333-bccc-e1c7b34286c7-plugins-config-volume" pod "21bd8ee2-cb78-4333-bccc-e1c7b34286c7" (UID: "21bd8ee2-cb78-4333-bccc-e1c7b34286c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:14.073526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.073483    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21bd8ee2-cb78-4333-bccc-e1c7b34286c7-kube-api-access-2j96p" pod "21bd8ee2-cb78-4333-bccc-e1c7b34286c7" (UID: "21bd8ee2-cb78-4333-bccc-e1c7b34286c7"). InnerVolumeSpecName "kube-api-access-2j96p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:14.082954 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.082650    5893 scope.go:122] "RemoveContainer" containerID="4e3f93b8d5d6cdceac21b5777955ad85c5a4377c93d5c4c72d25e1074e234db6"
Oct 30 13:39:14.083326 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:14.083196    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e3f93b8d5d6cdceac21b5777955ad85c5a4377c93d5c4c72d25e1074e234db6\": container with ID starting with 4e3f93b8d5d6cdceac21b5777955ad85c5a4377c93d5c4c72d25e1074e234db6 not found: ID does not exist" containerID="4e3f93b8d5d6cdceac21b5777955ad85c5a4377c93d5c4c72d25e1074e234db6"
Oct 30 13:39:14.083326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.083241    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e3f93b8d5d6cdceac21b5777955ad85c5a4377c93d5c4c72d25e1074e234db6"} err="failed to get container status \"4e3f93b8d5d6cdceac21b5777955ad85c5a4377c93d5c4c72d25e1074e234db6\": rpc error: code = NotFound desc = could not find container \"4e3f93b8d5d6cdceac21b5777955ad85c5a4377c93d5c4c72d25e1074e234db6\": container with ID starting with 4e3f93b8d5d6cdceac21b5777955ad85c5a4377c93d5c4c72d25e1074e234db6 not found: ID does not exist"
Oct 30 13:39:14.104306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.104237    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"]
Oct 30 13:39:14.111221 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.111192    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c91909-574f7f7cf8-2v2js"]
Oct 30 13:39:14.156251 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.156211    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j96p\" (UniqueName: \"kubernetes.io/projected/21bd8ee2-cb78-4333-bccc-e1c7b34286c7-kube-api-access-2j96p\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:14.156251 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.156251    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21bd8ee2-cb78-4333-bccc-e1c7b34286c7-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:14.363227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.363142    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"]
Oct 30 13:39:14.378340 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.373622    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c91909-6694cc5b8c-vcbvj"]
Oct 30 13:39:14.536842 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.536796    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21bd8ee2-cb78-4333-bccc-e1c7b34286c7" path="/var/lib/kubelet/pods/21bd8ee2-cb78-4333-bccc-e1c7b34286c7/volumes"
Oct 30 13:39:14.537575 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:14.537553    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd8433a-82e7-43f7-a036-1c79a64230a1" path="/var/lib/kubelet/pods/ddd8433a-82e7-43f7-a036-1c79a64230a1/volumes"
Oct 30 13:39:16.599382 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.599350    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:39:16.615673 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.615651    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:39:16.697186 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.697117    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-workload-socket\") pod \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") "
Oct 30 13:39:16.699703 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.699671    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-token\" (UniqueName: \"kubernetes.io/projected/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-token\") pod \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") "
Oct 30 13:39:16.699868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.699720    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-workload-certs\") pod \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") "
Oct 30 13:39:16.699868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.699806    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-data\") pod \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") "
Oct 30 13:39:16.699868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.699846    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-credential-socket\") pod \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") "
Oct 30 13:39:16.700062 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.699883    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-token\" (UniqueName: \"kubernetes.io/projected/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-token\") pod \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") "
Oct 30 13:39:16.700062 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.699913    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-podinfo\") pod \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") "
Oct 30 13:39:16.700062 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.699983    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-data\") pod \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") "
Oct 30 13:39:16.700062 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.700024    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-podinfo\") pod \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") "
Oct 30 13:39:16.700580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.700125    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-credential-socket\") pod \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") "
Oct 30 13:39:16.700580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.700200    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-envoy\") pod \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") "
Oct 30 13:39:16.700580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.700252    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istiod-ca-cert\") pod \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") "
Oct 30 13:39:16.706405 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.698501    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-workload-socket" pod "805ec71c-0b0c-48a9-a154-7fbf568f73ba" (UID: "805ec71c-0b0c-48a9-a154-7fbf568f73ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:16.706518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.706128    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-credential-socket" pod "805ec71c-0b0c-48a9-a154-7fbf568f73ba" (UID: "805ec71c-0b0c-48a9-a154-7fbf568f73ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:16.706518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.706378    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-workload-certs" pod "721c5b17-8eeb-4e8b-94b4-2d46a18a384c" (UID: "721c5b17-8eeb-4e8b-94b4-2d46a18a384c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:16.706848 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.706824    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-data" pod "805ec71c-0b0c-48a9-a154-7fbf568f73ba" (UID: "805ec71c-0b0c-48a9-a154-7fbf568f73ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:16.707038 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.707020    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-credential-socket" pod "721c5b17-8eeb-4e8b-94b4-2d46a18a384c" (UID: "721c5b17-8eeb-4e8b-94b4-2d46a18a384c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:16.707622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.707181    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-data" pod "721c5b17-8eeb-4e8b-94b4-2d46a18a384c" (UID: "721c5b17-8eeb-4e8b-94b4-2d46a18a384c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:16.707732 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.707720    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istiod-ca-cert" pod "721c5b17-8eeb-4e8b-94b4-2d46a18a384c" (UID: "721c5b17-8eeb-4e8b-94b4-2d46a18a384c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:16.707900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.707870    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-envoy\") pod \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") "
Oct 30 13:39:16.708670 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.708650    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-kube-api-access-mlfw7\" (UniqueName: \"kubernetes.io/projected/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-kube-api-access-mlfw7\") pod \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") "
Oct 30 13:39:16.708815 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.708801    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/805ec71c-0b0c-48a9-a154-7fbf568f73ba-kube-api-access-vc6wv\" (UniqueName: \"kubernetes.io/projected/805ec71c-0b0c-48a9-a154-7fbf568f73ba-kube-api-access-vc6wv\") pod \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") "
Oct 30 13:39:16.708980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.708967    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istiod-ca-cert\") pod \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") "
Oct 30 13:39:16.709094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.709071    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-workload-socket\") pod \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\" (UID: \"721c5b17-8eeb-4e8b-94b4-2d46a18a384c\") "
Oct 30 13:39:16.709203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.709105    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-workload-certs\") pod \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\" (UID: \"805ec71c-0b0c-48a9-a154-7fbf568f73ba\") "
Oct 30 13:39:16.709697 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.709668    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istiod-ca-cert" pod "805ec71c-0b0c-48a9-a154-7fbf568f73ba" (UID: "805ec71c-0b0c-48a9-a154-7fbf568f73ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:16.710031 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.710013    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.710674 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.710659    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.710816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.710804    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.710995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.710913    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.711118 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.711106    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.711217 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.711207    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.711338 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.711325    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.711452 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.711434    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.711452 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.710220    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-podinfo" pod "721c5b17-8eeb-4e8b-94b4-2d46a18a384c" (UID: "721c5b17-8eeb-4e8b-94b4-2d46a18a384c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:16.711699 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.710463    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-workload-socket" pod "721c5b17-8eeb-4e8b-94b4-2d46a18a384c" (UID: "721c5b17-8eeb-4e8b-94b4-2d46a18a384c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:16.711699 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.710621    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-workload-certs" pod "805ec71c-0b0c-48a9-a154-7fbf568f73ba" (UID: "805ec71c-0b0c-48a9-a154-7fbf568f73ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:16.712663 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.712619    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-podinfo" pod "805ec71c-0b0c-48a9-a154-7fbf568f73ba" (UID: "805ec71c-0b0c-48a9-a154-7fbf568f73ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:16.712790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.712770    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-token" pod "721c5b17-8eeb-4e8b-94b4-2d46a18a384c" (UID: "721c5b17-8eeb-4e8b-94b4-2d46a18a384c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:16.712866 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.712854    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-kube-api-access-mlfw7" pod "721c5b17-8eeb-4e8b-94b4-2d46a18a384c" (UID: "721c5b17-8eeb-4e8b-94b4-2d46a18a384c"). InnerVolumeSpecName "kube-api-access-mlfw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:16.712961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.712943    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-envoy" pod "805ec71c-0b0c-48a9-a154-7fbf568f73ba" (UID: "805ec71c-0b0c-48a9-a154-7fbf568f73ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:16.713256 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.713059    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-token" pod "805ec71c-0b0c-48a9-a154-7fbf568f73ba" (UID: "805ec71c-0b0c-48a9-a154-7fbf568f73ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:16.713754 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.713735    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/805ec71c-0b0c-48a9-a154-7fbf568f73ba-kube-api-access-vc6wv" pod "805ec71c-0b0c-48a9-a154-7fbf568f73ba" (UID: "805ec71c-0b0c-48a9-a154-7fbf568f73ba"). InnerVolumeSpecName "kube-api-access-vc6wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:16.714914 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.714896    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-envoy" pod "721c5b17-8eeb-4e8b-94b4-2d46a18a384c" (UID: "721c5b17-8eeb-4e8b-94b4-2d46a18a384c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:16.813092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.813003    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlfw7\" (UniqueName: \"kubernetes.io/projected/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-kube-api-access-mlfw7\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.813092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.813041    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc6wv\" (UniqueName: \"kubernetes.io/projected/805ec71c-0b0c-48a9-a154-7fbf568f73ba-kube-api-access-vc6wv\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.813092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.813057    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.813092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.813071    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.813092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.813085    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.813403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.813097    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.813403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.813109    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.813403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.813121    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.813403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.813133    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/805ec71c-0b0c-48a9-a154-7fbf568f73ba-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:16.813403 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:16.813145    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/721c5b17-8eeb-4e8b-94b4-2d46a18a384c-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:17.036318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.036264    5893 generic.go:356] "Generic (PLEG): container finished" podID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerID="be190cf826b3356ece2e753fe3e0efc84bfd8139859a57b0765d2ceb1b78ed76" exitCode=0
Oct 30 13:39:17.036505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.036462    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql" event={"ID":"721c5b17-8eeb-4e8b-94b4-2d46a18a384c","Type":"ContainerDied","Data":"be190cf826b3356ece2e753fe3e0efc84bfd8139859a57b0765d2ceb1b78ed76"}
Oct 30 13:39:17.036566 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.036552    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql" event={"ID":"721c5b17-8eeb-4e8b-94b4-2d46a18a384c","Type":"ContainerDied","Data":"260e6f0bbf4dc92e3621742e2023f567e8a4e9e4cf66b7e1cc9743d6858fa40f"}
Oct 30 13:39:17.036612 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.036587    5893 scope.go:122] "RemoveContainer" containerID="be190cf826b3356ece2e753fe3e0efc84bfd8139859a57b0765d2ceb1b78ed76"
Oct 30 13:39:17.036758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.036744    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"
Oct 30 13:39:17.047461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.047427    5893 generic.go:356] "Generic (PLEG): container finished" podID="805ec71c-0b0c-48a9-a154-7fbf568f73ba" containerID="cc5765085692b031c63afac12f8909598aab3f7a7fc7486109de03abad941fcd" exitCode=0
Oct 30 13:39:17.047596 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.047521    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"
Oct 30 13:39:17.047657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.047615    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj" event={"ID":"805ec71c-0b0c-48a9-a154-7fbf568f73ba","Type":"ContainerDied","Data":"cc5765085692b031c63afac12f8909598aab3f7a7fc7486109de03abad941fcd"}
Oct 30 13:39:17.047657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.047654    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj" event={"ID":"805ec71c-0b0c-48a9-a154-7fbf568f73ba","Type":"ContainerDied","Data":"d23cafc125802c919ba71ce6c9dd4a3dc8d59505106b12c757aa7226aba667aa"}
Oct 30 13:39:17.059386 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.059366    5893 scope.go:122] "RemoveContainer" containerID="fc9e6ffdc69314a52f0184a5f4dc0faa824e7f7d0642a236f71620c0051ede81"
Oct 30 13:39:17.074263 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.074231    5893 scope.go:122] "RemoveContainer" containerID="be190cf826b3356ece2e753fe3e0efc84bfd8139859a57b0765d2ceb1b78ed76"
Oct 30 13:39:17.075306 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:17.075241    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be190cf826b3356ece2e753fe3e0efc84bfd8139859a57b0765d2ceb1b78ed76\": container with ID starting with be190cf826b3356ece2e753fe3e0efc84bfd8139859a57b0765d2ceb1b78ed76 not found: ID does not exist" containerID="be190cf826b3356ece2e753fe3e0efc84bfd8139859a57b0765d2ceb1b78ed76"
Oct 30 13:39:17.075430 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.075307    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be190cf826b3356ece2e753fe3e0efc84bfd8139859a57b0765d2ceb1b78ed76"} err="failed to get container status \"be190cf826b3356ece2e753fe3e0efc84bfd8139859a57b0765d2ceb1b78ed76\": rpc error: code = NotFound desc = could not find container \"be190cf826b3356ece2e753fe3e0efc84bfd8139859a57b0765d2ceb1b78ed76\": container with ID starting with be190cf826b3356ece2e753fe3e0efc84bfd8139859a57b0765d2ceb1b78ed76 not found: ID does not exist"
Oct 30 13:39:17.075430 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.075327    5893 scope.go:122] "RemoveContainer" containerID="fc9e6ffdc69314a52f0184a5f4dc0faa824e7f7d0642a236f71620c0051ede81"
Oct 30 13:39:17.075678 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:17.075654    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9e6ffdc69314a52f0184a5f4dc0faa824e7f7d0642a236f71620c0051ede81\": container with ID starting with fc9e6ffdc69314a52f0184a5f4dc0faa824e7f7d0642a236f71620c0051ede81 not found: ID does not exist" containerID="fc9e6ffdc69314a52f0184a5f4dc0faa824e7f7d0642a236f71620c0051ede81"
Oct 30 13:39:17.075678 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.075681    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9e6ffdc69314a52f0184a5f4dc0faa824e7f7d0642a236f71620c0051ede81"} err="failed to get container status \"fc9e6ffdc69314a52f0184a5f4dc0faa824e7f7d0642a236f71620c0051ede81\": rpc error: code = NotFound desc = could not find container \"fc9e6ffdc69314a52f0184a5f4dc0faa824e7f7d0642a236f71620c0051ede81\": container with ID starting with fc9e6ffdc69314a52f0184a5f4dc0faa824e7f7d0642a236f71620c0051ede81 not found: ID does not exist"
Oct 30 13:39:17.075878 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.075702    5893 scope.go:122] "RemoveContainer" containerID="cc5765085692b031c63afac12f8909598aab3f7a7fc7486109de03abad941fcd"
Oct 30 13:39:17.090889 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.090123    5893 scope.go:122] "RemoveContainer" containerID="cc5765085692b031c63afac12f8909598aab3f7a7fc7486109de03abad941fcd"
Oct 30 13:39:17.090889 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:17.090558    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc5765085692b031c63afac12f8909598aab3f7a7fc7486109de03abad941fcd\": container with ID starting with cc5765085692b031c63afac12f8909598aab3f7a7fc7486109de03abad941fcd not found: ID does not exist" containerID="cc5765085692b031c63afac12f8909598aab3f7a7fc7486109de03abad941fcd"
Oct 30 13:39:17.090889 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.090594    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc5765085692b031c63afac12f8909598aab3f7a7fc7486109de03abad941fcd"} err="failed to get container status \"cc5765085692b031c63afac12f8909598aab3f7a7fc7486109de03abad941fcd\": rpc error: code = NotFound desc = could not find container \"cc5765085692b031c63afac12f8909598aab3f7a7fc7486109de03abad941fcd\": container with ID starting with cc5765085692b031c63afac12f8909598aab3f7a7fc7486109de03abad941fcd not found: ID does not exist"
Oct 30 13:39:17.101306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.101247    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"]
Oct 30 13:39:17.105899 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.105875    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-mxcql"]
Oct 30 13:39:17.136058 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.136032    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"]
Oct 30 13:39:17.136263 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.136243    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6" podUID="b75c100c-a950-4e30-9c47-51ec63a8b09a" containerName="maas-api" containerID="cri-o://6c4baa5ca1c1ef7e35ce9a25cc1e284c7034c5b3e92287125b1adf8e945546a5" gracePeriod=30
Oct 30 13:39:17.141861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.141843    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"]
Oct 30 13:39:17.158077 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.158045    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-8a5d5e-openshift-default-68946d7c9c-qs9lj"]
Oct 30 13:39:17.424871 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.424839    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 13:39:17.523644 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.523614    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b75c100c-a950-4e30-9c47-51ec63a8b09a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75c100c-a950-4e30-9c47-51ec63a8b09a-openshift-service-ca\") pod \"b75c100c-a950-4e30-9c47-51ec63a8b09a\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") "
Oct 30 13:39:17.523864 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.523670    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b75c100c-a950-4e30-9c47-51ec63a8b09a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75c100c-a950-4e30-9c47-51ec63a8b09a-maas-api-tls\") pod \"b75c100c-a950-4e30-9c47-51ec63a8b09a\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") "
Oct 30 13:39:17.523864 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.523786    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75c100c-a950-4e30-9c47-51ec63a8b09a-kube-api-access-2td8l\" (UniqueName: \"kubernetes.io/projected/b75c100c-a950-4e30-9c47-51ec63a8b09a-kube-api-access-2td8l\") pod \"b75c100c-a950-4e30-9c47-51ec63a8b09a\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") "
Oct 30 13:39:17.523864 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.523822    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b75c100c-a950-4e30-9c47-51ec63a8b09a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75c100c-a950-4e30-9c47-51ec63a8b09a-metrics-certs\") pod \"b75c100c-a950-4e30-9c47-51ec63a8b09a\" (UID: \"b75c100c-a950-4e30-9c47-51ec63a8b09a\") "
Oct 30 13:39:17.524710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.524676    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75c100c-a950-4e30-9c47-51ec63a8b09a-openshift-service-ca" pod "b75c100c-a950-4e30-9c47-51ec63a8b09a" (UID: "b75c100c-a950-4e30-9c47-51ec63a8b09a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:17.525530 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.525287    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b75c100c-a950-4e30-9c47-51ec63a8b09a-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:17.527976 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.527937    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75c100c-a950-4e30-9c47-51ec63a8b09a-metrics-certs" pod "b75c100c-a950-4e30-9c47-51ec63a8b09a" (UID: "b75c100c-a950-4e30-9c47-51ec63a8b09a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:17.527976 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.527942    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75c100c-a950-4e30-9c47-51ec63a8b09a-kube-api-access-2td8l" pod "b75c100c-a950-4e30-9c47-51ec63a8b09a" (UID: "b75c100c-a950-4e30-9c47-51ec63a8b09a"). InnerVolumeSpecName "kube-api-access-2td8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:17.528135 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.528008    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75c100c-a950-4e30-9c47-51ec63a8b09a-maas-api-tls" pod "b75c100c-a950-4e30-9c47-51ec63a8b09a" (UID: "b75c100c-a950-4e30-9c47-51ec63a8b09a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:17.646353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.629431    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b75c100c-a950-4e30-9c47-51ec63a8b09a-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:17.646353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.629471    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2td8l\" (UniqueName: \"kubernetes.io/projected/b75c100c-a950-4e30-9c47-51ec63a8b09a-kube-api-access-2td8l\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:17.646353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.629486    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b75c100c-a950-4e30-9c47-51ec63a8b09a-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:17.773501 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.769825    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"]
Oct 30 13:39:17.773501 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.770057    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5" podUID="54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" containerName="istio-proxy" containerID="cri-o://8c1ba04c65ab34c2495913a5ec3b070904f7eaacbe28b282d22cbe0f6d076f09" gracePeriod=30
Oct 30 13:39:17.837099 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.837061    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z" podUID="2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f" containerName="payload-processing" containerID="cri-o://b1bb6868e44657b01709e7e68add5abc6f877bd3e37ace3ec13af324d849248e" gracePeriod=25
Oct 30 13:39:17.838971 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.838945    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"]
Oct 30 13:39:17.839208 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.839185    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" podUID="98d3dc24-762e-4470-a524-c5d7c822ea80" containerName="istio-proxy" containerID="cri-o://5ba9225218b01720724f795a15a11d88be7856e25333d61294a7e1f45c3743eb" gracePeriod=30
Oct 30 13:39:17.882214 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:17.882177    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j" podUID="09976b18-da28-4c48-8ac1-c3f914e632b4" containerName="payload-pre-processing" containerID="cri-o://6c8a13ecaaa729cc1852e867e97f9951a01e8fc7e322cd278f454424c3ff1af7" gracePeriod=25
Oct 30 13:39:18.063685 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.063647    5893 generic.go:356] "Generic (PLEG): container finished" podID="09976b18-da28-4c48-8ac1-c3f914e632b4" containerID="6c8a13ecaaa729cc1852e867e97f9951a01e8fc7e322cd278f454424c3ff1af7" exitCode=0
Oct 30 13:39:18.063882 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.063798    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j" event={"ID":"09976b18-da28-4c48-8ac1-c3f914e632b4","Type":"ContainerDied","Data":"6c8a13ecaaa729cc1852e867e97f9951a01e8fc7e322cd278f454424c3ff1af7"}
Oct 30 13:39:18.068196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.067944    5893 generic.go:356] "Generic (PLEG): container finished" podID="b75c100c-a950-4e30-9c47-51ec63a8b09a" containerID="6c4baa5ca1c1ef7e35ce9a25cc1e284c7034c5b3e92287125b1adf8e945546a5" exitCode=0
Oct 30 13:39:18.068196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.067996    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"
Oct 30 13:39:18.068196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.068052    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6" event={"ID":"b75c100c-a950-4e30-9c47-51ec63a8b09a","Type":"ContainerDied","Data":"6c4baa5ca1c1ef7e35ce9a25cc1e284c7034c5b3e92287125b1adf8e945546a5"}
Oct 30 13:39:18.068196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.068092    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6" event={"ID":"b75c100c-a950-4e30-9c47-51ec63a8b09a","Type":"ContainerDied","Data":"ed6a8e8242bd0f82792c6f9f7f0c278560e34f92a00495210485fc07dadbdf43"}
Oct 30 13:39:18.068196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.068119    5893 scope.go:122] "RemoveContainer" containerID="6c4baa5ca1c1ef7e35ce9a25cc1e284c7034c5b3e92287125b1adf8e945546a5"
Oct 30 13:39:18.075344 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.074045    5893 generic.go:356] "Generic (PLEG): container finished" podID="2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f" containerID="b1bb6868e44657b01709e7e68add5abc6f877bd3e37ace3ec13af324d849248e" exitCode=0
Oct 30 13:39:18.075344 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.074196    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z" event={"ID":"2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f","Type":"ContainerDied","Data":"b1bb6868e44657b01709e7e68add5abc6f877bd3e37ace3ec13af324d849248e"}
Oct 30 13:39:18.091233 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.091212    5893 scope.go:122] "RemoveContainer" containerID="6c4baa5ca1c1ef7e35ce9a25cc1e284c7034c5b3e92287125b1adf8e945546a5"
Oct 30 13:39:18.091816 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:18.091791    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c4baa5ca1c1ef7e35ce9a25cc1e284c7034c5b3e92287125b1adf8e945546a5\": container with ID starting with 6c4baa5ca1c1ef7e35ce9a25cc1e284c7034c5b3e92287125b1adf8e945546a5 not found: ID does not exist" containerID="6c4baa5ca1c1ef7e35ce9a25cc1e284c7034c5b3e92287125b1adf8e945546a5"
Oct 30 13:39:18.091934 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.091828    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c4baa5ca1c1ef7e35ce9a25cc1e284c7034c5b3e92287125b1adf8e945546a5"} err="failed to get container status \"6c4baa5ca1c1ef7e35ce9a25cc1e284c7034c5b3e92287125b1adf8e945546a5\": rpc error: code = NotFound desc = could not find container \"6c4baa5ca1c1ef7e35ce9a25cc1e284c7034c5b3e92287125b1adf8e945546a5\": container with ID starting with 6c4baa5ca1c1ef7e35ce9a25cc1e284c7034c5b3e92287125b1adf8e945546a5 not found: ID does not exist"
Oct 30 13:39:18.130244 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.128310    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"]
Oct 30 13:39:18.132451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.132430    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-09db16-57b9d7cc7c-mmkq6"]
Oct 30 13:39:18.290056 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.290004    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-5tgdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:39:18.290353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.290320    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" podUID="98d3dc24-762e-4470-a524-c5d7c822ea80" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:39:18.294865 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.294844    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"
Oct 30 13:39:18.322153 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.322133    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"
Oct 30 13:39:18.367225 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.367179    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f-kube-api-access-5fppd\" (UniqueName: \"kubernetes.io/projected/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f-kube-api-access-5fppd\") pod \"2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f\" (UID: \"2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f\") "
Oct 30 13:39:18.367462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.367262    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f-plugins-config-volume\") pod \"2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f\" (UID: \"2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f\") "
Oct 30 13:39:18.368095 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.368066    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f-plugins-config-volume" pod "2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f" (UID: "2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:18.376808 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.376772    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f-kube-api-access-5fppd" pod "2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f" (UID: "2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f"). InnerVolumeSpecName "kube-api-access-5fppd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:18.469144 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.469114    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09976b18-da28-4c48-8ac1-c3f914e632b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09976b18-da28-4c48-8ac1-c3f914e632b4-plugins-config-volume\") pod \"09976b18-da28-4c48-8ac1-c3f914e632b4\" (UID: \"09976b18-da28-4c48-8ac1-c3f914e632b4\") "
Oct 30 13:39:18.469377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.469202    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09976b18-da28-4c48-8ac1-c3f914e632b4-kube-api-access-cbmlp\" (UniqueName: \"kubernetes.io/projected/09976b18-da28-4c48-8ac1-c3f914e632b4-kube-api-access-cbmlp\") pod \"09976b18-da28-4c48-8ac1-c3f914e632b4\" (UID: \"09976b18-da28-4c48-8ac1-c3f914e632b4\") "
Oct 30 13:39:18.469522 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.469499    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09976b18-da28-4c48-8ac1-c3f914e632b4-plugins-config-volume" pod "09976b18-da28-4c48-8ac1-c3f914e632b4" (UID: "09976b18-da28-4c48-8ac1-c3f914e632b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:18.469738 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.469723    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fppd\" (UniqueName: \"kubernetes.io/projected/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f-kube-api-access-5fppd\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:18.469797 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.469740    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:18.469797 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.469749    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/09976b18-da28-4c48-8ac1-c3f914e632b4-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:18.471549 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.471525    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09976b18-da28-4c48-8ac1-c3f914e632b4-kube-api-access-cbmlp" pod "09976b18-da28-4c48-8ac1-c3f914e632b4" (UID: "09976b18-da28-4c48-8ac1-c3f914e632b4"). InnerVolumeSpecName "kube-api-access-cbmlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:18.533543 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.533478    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" path="/var/lib/kubelet/pods/721c5b17-8eeb-4e8b-94b4-2d46a18a384c/volumes"
Oct 30 13:39:18.534053 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.534029    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="805ec71c-0b0c-48a9-a154-7fbf568f73ba" path="/var/lib/kubelet/pods/805ec71c-0b0c-48a9-a154-7fbf568f73ba/volumes"
Oct 30 13:39:18.534580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.534565    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b75c100c-a950-4e30-9c47-51ec63a8b09a" path="/var/lib/kubelet/pods/b75c100c-a950-4e30-9c47-51ec63a8b09a/volumes"
Oct 30 13:39:18.570980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:18.570956    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbmlp\" (UniqueName: \"kubernetes.io/projected/09976b18-da28-4c48-8ac1-c3f914e632b4-kube-api-access-cbmlp\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:18.646679 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:18.646641    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:39:19.084163 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:19.084137    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"
Oct 30 13:39:19.084363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:19.084162    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j" event={"ID":"09976b18-da28-4c48-8ac1-c3f914e632b4","Type":"ContainerDied","Data":"3b693caeb0bc02f47a1aed8307312a70f29ca1de3bc459a959860d2dd49128c7"}
Oct 30 13:39:19.084363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:19.084232    5893 scope.go:122] "RemoveContainer" containerID="6c8a13ecaaa729cc1852e867e97f9951a01e8fc7e322cd278f454424c3ff1af7"
Oct 30 13:39:19.087013 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:19.086992    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"
Oct 30 13:39:19.087136 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:19.087031    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z" event={"ID":"2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f","Type":"ContainerDied","Data":"e3b31544de5cc5d4cd64227908a0f04dab5eb5f2f89aae1a090926bbd6437652"}
Oct 30 13:39:19.097439 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:19.097421    5893 scope.go:122] "RemoveContainer" containerID="b1bb6868e44657b01709e7e68add5abc6f877bd3e37ace3ec13af324d849248e"
Oct 30 13:39:19.133050 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:19.133023    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"]
Oct 30 13:39:19.140710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:19.140682    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-6c7dc8f7c9-dpk5j"]
Oct 30 13:39:19.175700 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:19.175670    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"]
Oct 30 13:39:19.178395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:19.178369    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba8145-564d8b96b6-5s56z"]
Oct 30 13:39:20.542172 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:20.542132    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09976b18-da28-4c48-8ac1-c3f914e632b4" path="/var/lib/kubelet/pods/09976b18-da28-4c48-8ac1-c3f914e632b4/volumes"
Oct 30 13:39:20.542807 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:20.542770    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f" path="/var/lib/kubelet/pods/2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f/volumes"
Oct 30 13:39:23.143175 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.143098    5893 generic.go:356] "Generic (PLEG): container finished" podID="98d3dc24-762e-4470-a524-c5d7c822ea80" containerID="5ba9225218b01720724f795a15a11d88be7856e25333d61294a7e1f45c3743eb" exitCode=0
Oct 30 13:39:23.143646 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.143341    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" event={"ID":"98d3dc24-762e-4470-a524-c5d7c822ea80","Type":"ContainerDied","Data":"5ba9225218b01720724f795a15a11d88be7856e25333d61294a7e1f45c3743eb"}
Oct 30 13:39:23.145338 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.145311    5893 generic.go:356] "Generic (PLEG): container finished" podID="54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" containerID="8c1ba04c65ab34c2495913a5ec3b070904f7eaacbe28b282d22cbe0f6d076f09" exitCode=0
Oct 30 13:39:23.145457 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.145404    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5" event={"ID":"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb","Type":"ContainerDied","Data":"8c1ba04c65ab34c2495913a5ec3b070904f7eaacbe28b282d22cbe0f6d076f09"}
Oct 30 13:39:23.241529 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.241434    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 13:39:23.275639 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.275525    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:39:23.350132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.350094    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-envoy\") pod \"98d3dc24-762e-4470-a524-c5d7c822ea80\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") "
Oct 30 13:39:23.350391 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.350193    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-workload-certs\") pod \"98d3dc24-762e-4470-a524-c5d7c822ea80\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") "
Oct 30 13:39:23.350391 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.350255    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98d3dc24-762e-4470-a524-c5d7c822ea80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d3dc24-762e-4470-a524-c5d7c822ea80-istiod-ca-cert\") pod \"98d3dc24-762e-4470-a524-c5d7c822ea80\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") "
Oct 30 13:39:23.350391 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.350328    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-data\") pod \"98d3dc24-762e-4470-a524-c5d7c822ea80\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") "
Oct 30 13:39:23.353905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.350415    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-workload-socket\") pod \"98d3dc24-762e-4470-a524-c5d7c822ea80\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") "
Oct 30 13:39:23.353905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.350444    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-podinfo\") pod \"98d3dc24-762e-4470-a524-c5d7c822ea80\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") "
Oct 30 13:39:23.353905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.350503    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d3dc24-762e-4470-a524-c5d7c822ea80-kube-api-access-cbrtl\" (UniqueName: \"kubernetes.io/projected/98d3dc24-762e-4470-a524-c5d7c822ea80-kube-api-access-cbrtl\") pod \"98d3dc24-762e-4470-a524-c5d7c822ea80\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") "
Oct 30 13:39:23.353905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.350618    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-token\" (UniqueName: \"kubernetes.io/projected/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-token\") pod \"98d3dc24-762e-4470-a524-c5d7c822ea80\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") "
Oct 30 13:39:23.353905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.350673    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-credential-socket\") pod \"98d3dc24-762e-4470-a524-c5d7c822ea80\" (UID: \"98d3dc24-762e-4470-a524-c5d7c822ea80\") "
Oct 30 13:39:23.353905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.351062    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-data" pod "98d3dc24-762e-4470-a524-c5d7c822ea80" (UID: "98d3dc24-762e-4470-a524-c5d7c822ea80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:23.353905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.351320    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.353905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.351326    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-workload-socket" pod "98d3dc24-762e-4470-a524-c5d7c822ea80" (UID: "98d3dc24-762e-4470-a524-c5d7c822ea80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:23.353905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.351618    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98d3dc24-762e-4470-a524-c5d7c822ea80-istiod-ca-cert" pod "98d3dc24-762e-4470-a524-c5d7c822ea80" (UID: "98d3dc24-762e-4470-a524-c5d7c822ea80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:23.353905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.351731    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-workload-certs" pod "98d3dc24-762e-4470-a524-c5d7c822ea80" (UID: "98d3dc24-762e-4470-a524-c5d7c822ea80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:23.353905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.351841    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-credential-socket" pod "98d3dc24-762e-4470-a524-c5d7c822ea80" (UID: "98d3dc24-762e-4470-a524-c5d7c822ea80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:23.355295 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.355226    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-podinfo" pod "98d3dc24-762e-4470-a524-c5d7c822ea80" (UID: "98d3dc24-762e-4470-a524-c5d7c822ea80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:23.355415 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.355310    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-token" pod "98d3dc24-762e-4470-a524-c5d7c822ea80" (UID: "98d3dc24-762e-4470-a524-c5d7c822ea80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:23.355415 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.355340    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98d3dc24-762e-4470-a524-c5d7c822ea80-kube-api-access-cbrtl" pod "98d3dc24-762e-4470-a524-c5d7c822ea80" (UID: "98d3dc24-762e-4470-a524-c5d7c822ea80"). InnerVolumeSpecName "kube-api-access-cbrtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:23.357365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.357322    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-envoy" pod "98d3dc24-762e-4470-a524-c5d7c822ea80" (UID: "98d3dc24-762e-4470-a524-c5d7c822ea80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:23.452836 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.452743    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-data\") pod \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") "
Oct 30 13:39:23.452836 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.452804    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-credential-socket\") pod \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") "
Oct 30 13:39:23.453095 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.452967    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-envoy\") pod \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") "
Oct 30 13:39:23.453095 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.453045    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-token\" (UniqueName: \"kubernetes.io/projected/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-token\") pod \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") "
Oct 30 13:39:23.453095 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.453048    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-data" pod "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" (UID: "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:23.453267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.453102    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-workload-certs\") pod \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") "
Oct 30 13:39:23.453267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.453154    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-credential-socket" pod "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" (UID: "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:23.453267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.453163    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-kube-api-access-wsjkp\" (UniqueName: \"kubernetes.io/projected/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-kube-api-access-wsjkp\") pod \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") "
Oct 30 13:39:23.453267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.453201    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-workload-socket\") pod \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") "
Oct 30 13:39:23.453267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.453243    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istiod-ca-cert\") pod \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") "
Oct 30 13:39:23.453651 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.453318    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-podinfo\") pod \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\" (UID: \"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb\") "
Oct 30 13:39:23.453651 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.453435    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-workload-certs" pod "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" (UID: "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:23.453651 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.453598    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-workload-socket" pod "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" (UID: "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:23.453822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.453713    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istiod-ca-cert" pod "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" (UID: "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:23.453998 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.453976    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.454363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.454001    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.454363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.454015    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.454363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.454028    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.454363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.454044    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.454363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.454056    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.454363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.454069    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.454363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.454083    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.454363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.454096    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98d3dc24-762e-4470-a524-c5d7c822ea80-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.454363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.454108    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98d3dc24-762e-4470-a524-c5d7c822ea80-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.454363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.454122    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98d3dc24-762e-4470-a524-c5d7c822ea80-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.454363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.454138    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbrtl\" (UniqueName: \"kubernetes.io/projected/98d3dc24-762e-4470-a524-c5d7c822ea80-kube-api-access-cbrtl\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.454363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.454151    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.455956 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.455913    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-kube-api-access-wsjkp" pod "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" (UID: "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb"). InnerVolumeSpecName "kube-api-access-wsjkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:23.456016 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.455980    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-envoy" pod "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" (UID: "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:23.456116 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.456099    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-token" pod "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" (UID: "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:23.456226 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.456210    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-podinfo" pod "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" (UID: "54fb1fae-c51f-4f04-a6ae-110f42dfd9eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:23.555064 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.555028    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.555064 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.555057    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.555064 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.555072    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsjkp\" (UniqueName: \"kubernetes.io/projected/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-kube-api-access-wsjkp\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:23.555407 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:23.555086    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:24.155780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:24.155686    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"
Oct 30 13:39:24.155780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:24.155714    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5" event={"ID":"54fb1fae-c51f-4f04-a6ae-110f42dfd9eb","Type":"ContainerDied","Data":"5b8e245f857da7d3829135a995717c0125dc5ebcda8bb2be78c39916e20c750b"}
Oct 30 13:39:24.156321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:24.155793    5893 scope.go:122] "RemoveContainer" containerID="8c1ba04c65ab34c2495913a5ec3b070904f7eaacbe28b282d22cbe0f6d076f09"
Oct 30 13:39:24.157958 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:24.157935    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"
Oct 30 13:39:24.158113 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:24.157973    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl" event={"ID":"98d3dc24-762e-4470-a524-c5d7c822ea80","Type":"ContainerDied","Data":"65a9c20c3b38676e7b2d1c3b7f55e92aca7ee7edd411c120924d2dc86ffb8976"}
Oct 30 13:39:24.173806 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:24.173786    5893 scope.go:122] "RemoveContainer" containerID="5ba9225218b01720724f795a15a11d88be7856e25333d61294a7e1f45c3743eb"
Oct 30 13:39:24.225261 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:24.225214    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"]
Oct 30 13:39:24.234838 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:24.234808    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-5tgdl"]
Oct 30 13:39:24.263874 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:24.263847    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"]
Oct 30 13:39:24.267919 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:24.267890    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-acd125-openshift-default-688b6f7db7-k4zf5"]
Oct 30 13:39:24.539241 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:24.539198    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" path="/var/lib/kubelet/pods/54fb1fae-c51f-4f04-a6ae-110f42dfd9eb/volumes"
Oct 30 13:39:24.540045 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:24.540019    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98d3dc24-762e-4470-a524-c5d7c822ea80" path="/var/lib/kubelet/pods/98d3dc24-762e-4470-a524-c5d7c822ea80/volumes"
Oct 30 13:39:27.480917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:27.480885    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"]
Oct 30 13:39:27.481462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:27.481091    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4" podUID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerName="istio-proxy" containerID="cri-o://b36f78528c6901194e30801cf5f1aaa6e4c25fb3ce31f0d0f6c9a6bd7c148f3b" gracePeriod=30
Oct 30 13:39:28.935330 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:28.935285    5893 patch_prober.go:29] interesting pod/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:39:28.935806 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:28.935359    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4" podUID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:39:32.771535 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.771507    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 13:39:32.903031 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.903000    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-credential-socket\") pod \"ac623bd1-d3de-47f4-9e06-a2261a160df8\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") "
Oct 30 13:39:32.903218 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.903055    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-workload-socket\") pod \"ac623bd1-d3de-47f4-9e06-a2261a160df8\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") "
Oct 30 13:39:32.903218 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.903146    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-podinfo\") pod \"ac623bd1-d3de-47f4-9e06-a2261a160df8\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") "
Oct 30 13:39:32.903401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.903263    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-workload-certs\") pod \"ac623bd1-d3de-47f4-9e06-a2261a160df8\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") "
Oct 30 13:39:32.903401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.903355    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac623bd1-d3de-47f4-9e06-a2261a160df8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac623bd1-d3de-47f4-9e06-a2261a160df8-istiod-ca-cert\") pod \"ac623bd1-d3de-47f4-9e06-a2261a160df8\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") "
Oct 30 13:39:32.903401 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.903388    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac623bd1-d3de-47f4-9e06-a2261a160df8-kube-api-access-94ztk\" (UniqueName: \"kubernetes.io/projected/ac623bd1-d3de-47f4-9e06-a2261a160df8-kube-api-access-94ztk\") pod \"ac623bd1-d3de-47f4-9e06-a2261a160df8\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") "
Oct 30 13:39:32.903564 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.903438    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-token\" (UniqueName: \"kubernetes.io/projected/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-token\") pod \"ac623bd1-d3de-47f4-9e06-a2261a160df8\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") "
Oct 30 13:39:32.903564 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.903380    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-credential-socket" pod "ac623bd1-d3de-47f4-9e06-a2261a160df8" (UID: "ac623bd1-d3de-47f4-9e06-a2261a160df8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:32.903564 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.903390    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-workload-socket" pod "ac623bd1-d3de-47f4-9e06-a2261a160df8" (UID: "ac623bd1-d3de-47f4-9e06-a2261a160df8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:32.903564 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.903534    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-data\") pod \"ac623bd1-d3de-47f4-9e06-a2261a160df8\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") "
Oct 30 13:39:32.903972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.903590    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-envoy\") pod \"ac623bd1-d3de-47f4-9e06-a2261a160df8\" (UID: \"ac623bd1-d3de-47f4-9e06-a2261a160df8\") "
Oct 30 13:39:32.903972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.903588    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-workload-certs" pod "ac623bd1-d3de-47f4-9e06-a2261a160df8" (UID: "ac623bd1-d3de-47f4-9e06-a2261a160df8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:32.903972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.903709    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac623bd1-d3de-47f4-9e06-a2261a160df8-istiod-ca-cert" pod "ac623bd1-d3de-47f4-9e06-a2261a160df8" (UID: "ac623bd1-d3de-47f4-9e06-a2261a160df8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:32.903972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.903855    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-data" pod "ac623bd1-d3de-47f4-9e06-a2261a160df8" (UID: "ac623bd1-d3de-47f4-9e06-a2261a160df8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:32.904245 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.904227    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.904391 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.904253    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac623bd1-d3de-47f4-9e06-a2261a160df8-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.904454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.904401    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.904454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.904421    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.904535 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.904437    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:32.905822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.905791    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac623bd1-d3de-47f4-9e06-a2261a160df8-kube-api-access-94ztk" pod "ac623bd1-d3de-47f4-9e06-a2261a160df8" (UID: "ac623bd1-d3de-47f4-9e06-a2261a160df8"). InnerVolumeSpecName "kube-api-access-94ztk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:32.906032 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.906013    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-podinfo" pod "ac623bd1-d3de-47f4-9e06-a2261a160df8" (UID: "ac623bd1-d3de-47f4-9e06-a2261a160df8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:32.906359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.906338    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-envoy" pod "ac623bd1-d3de-47f4-9e06-a2261a160df8" (UID: "ac623bd1-d3de-47f4-9e06-a2261a160df8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:32.906469 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:32.906425    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-token" pod "ac623bd1-d3de-47f4-9e06-a2261a160df8" (UID: "ac623bd1-d3de-47f4-9e06-a2261a160df8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:33.006153 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.006076    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94ztk\" (UniqueName: \"kubernetes.io/projected/ac623bd1-d3de-47f4-9e06-a2261a160df8-kube-api-access-94ztk\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:33.006153 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.006103    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:33.006153 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.006114    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:33.006153 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.006122    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac623bd1-d3de-47f4-9e06-a2261a160df8-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:33.237083 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.237051    5893 generic.go:356] "Generic (PLEG): container finished" podID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerID="b36f78528c6901194e30801cf5f1aaa6e4c25fb3ce31f0d0f6c9a6bd7c148f3b" exitCode=0
Oct 30 13:39:33.237930 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.237105    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"
Oct 30 13:39:33.237930 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.237107    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4" event={"ID":"ac623bd1-d3de-47f4-9e06-a2261a160df8","Type":"ContainerDied","Data":"b36f78528c6901194e30801cf5f1aaa6e4c25fb3ce31f0d0f6c9a6bd7c148f3b"}
Oct 30 13:39:33.237930 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.237223    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4" event={"ID":"ac623bd1-d3de-47f4-9e06-a2261a160df8","Type":"ContainerDied","Data":"21fed8a66b85a6b7b5636ee1b455f82943bbe30b8f163ca76d5ac4f1e5bbcb65"}
Oct 30 13:39:33.237930 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.237246    5893 scope.go:122] "RemoveContainer" containerID="b36f78528c6901194e30801cf5f1aaa6e4c25fb3ce31f0d0f6c9a6bd7c148f3b"
Oct 30 13:39:33.249289 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.249256    5893 scope.go:122] "RemoveContainer" containerID="bfdc6246ef9576635f40aa600abe74f364919f6ff842637934def3dce6d5612c"
Oct 30 13:39:33.260287 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.260252    5893 scope.go:122] "RemoveContainer" containerID="b36f78528c6901194e30801cf5f1aaa6e4c25fb3ce31f0d0f6c9a6bd7c148f3b"
Oct 30 13:39:33.260611 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:33.260590    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36f78528c6901194e30801cf5f1aaa6e4c25fb3ce31f0d0f6c9a6bd7c148f3b\": container with ID starting with b36f78528c6901194e30801cf5f1aaa6e4c25fb3ce31f0d0f6c9a6bd7c148f3b not found: ID does not exist" containerID="b36f78528c6901194e30801cf5f1aaa6e4c25fb3ce31f0d0f6c9a6bd7c148f3b"
Oct 30 13:39:33.260675 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.260621    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36f78528c6901194e30801cf5f1aaa6e4c25fb3ce31f0d0f6c9a6bd7c148f3b"} err="failed to get container status \"b36f78528c6901194e30801cf5f1aaa6e4c25fb3ce31f0d0f6c9a6bd7c148f3b\": rpc error: code = NotFound desc = could not find container \"b36f78528c6901194e30801cf5f1aaa6e4c25fb3ce31f0d0f6c9a6bd7c148f3b\": container with ID starting with b36f78528c6901194e30801cf5f1aaa6e4c25fb3ce31f0d0f6c9a6bd7c148f3b not found: ID does not exist"
Oct 30 13:39:33.260675 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.260639    5893 scope.go:122] "RemoveContainer" containerID="bfdc6246ef9576635f40aa600abe74f364919f6ff842637934def3dce6d5612c"
Oct 30 13:39:33.260892 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:33.260874    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfdc6246ef9576635f40aa600abe74f364919f6ff842637934def3dce6d5612c\": container with ID starting with bfdc6246ef9576635f40aa600abe74f364919f6ff842637934def3dce6d5612c not found: ID does not exist" containerID="bfdc6246ef9576635f40aa600abe74f364919f6ff842637934def3dce6d5612c"
Oct 30 13:39:33.260931 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.260899    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfdc6246ef9576635f40aa600abe74f364919f6ff842637934def3dce6d5612c"} err="failed to get container status \"bfdc6246ef9576635f40aa600abe74f364919f6ff842637934def3dce6d5612c\": rpc error: code = NotFound desc = could not find container \"bfdc6246ef9576635f40aa600abe74f364919f6ff842637934def3dce6d5612c\": container with ID starting with bfdc6246ef9576635f40aa600abe74f364919f6ff842637934def3dce6d5612c not found: ID does not exist"
Oct 30 13:39:33.302860 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.302835    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"]
Oct 30 13:39:33.311586 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:33.311565    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-09db16-openshift-default-5d4f7c499-s4cr4"]
Oct 30 13:39:34.086602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:34.086568    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"]
Oct 30 13:39:34.532957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:34.532923    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac623bd1-d3de-47f4-9e06-a2261a160df8" path="/var/lib/kubelet/pods/ac623bd1-d3de-47f4-9e06-a2261a160df8/volumes"
Oct 30 13:39:34.863479 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:34.863387    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"]
Oct 30 13:39:34.902568 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:34.902525    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"]
Oct 30 13:39:39.087960 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:39.087897    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch" podUID="262cbe09-e912-4db9-b5ea-3d4ddbd073f5" containerName="payload-pre-processing" containerID="cri-o://cd199e634d6ac0a0db6ea45adec319b323e7cb36cdde3ccea9969cee1429041a" gracePeriod=25
Oct 30 13:39:39.864287 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:39.864219    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n" podUID="cccf12d4-0554-4ea8-95f3-ddf4e73317eb" containerName="payload-processing" containerID="cri-o://13fc22bd7cf83bf3b284c2cbfb78b7df10ebc66fec3b06eda29a8352c5925d5b" gracePeriod=25
Oct 30 13:39:39.903004 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:39.902964    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6" podUID="855e2bc2-43d6-4b6a-827f-a778178d6e23" containerName="payload-pre-processing" containerID="cri-o://d2d63accbeb2d6150c235f3a0f341989042f24d0ee9729eabb1c03b35193ded7" gracePeriod=25
Oct 30 13:39:40.207314 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.207288    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"
Oct 30 13:39:40.250480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.250456    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"
Oct 30 13:39:40.263914 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.263884    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"]
Oct 30 13:39:40.264170 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.264138    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t" podUID="1f632f65-19d6-4073-bafc-f840bb22ecc7" containerName="istio-proxy" containerID="cri-o://ab87ac6616b9b43c8df230fb3918f570953cbf1f474783af2aa3c621fcdafe80" gracePeriod=30
Oct 30 13:39:40.302633 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.302602    5893 generic.go:356] "Generic (PLEG): container finished" podID="855e2bc2-43d6-4b6a-827f-a778178d6e23" containerID="d2d63accbeb2d6150c235f3a0f341989042f24d0ee9729eabb1c03b35193ded7" exitCode=0
Oct 30 13:39:40.302831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.302660    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"
Oct 30 13:39:40.302831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.302724    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6" event={"ID":"855e2bc2-43d6-4b6a-827f-a778178d6e23","Type":"ContainerDied","Data":"d2d63accbeb2d6150c235f3a0f341989042f24d0ee9729eabb1c03b35193ded7"}
Oct 30 13:39:40.302831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.302773    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6" event={"ID":"855e2bc2-43d6-4b6a-827f-a778178d6e23","Type":"ContainerDied","Data":"7bf16658f7dc55d0e9a882191d0982f26038f90508b62dade31d74a9eb31fdac"}
Oct 30 13:39:40.302831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.302802    5893 scope.go:122] "RemoveContainer" containerID="d2d63accbeb2d6150c235f3a0f341989042f24d0ee9729eabb1c03b35193ded7"
Oct 30 13:39:40.304946 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.304915    5893 generic.go:356] "Generic (PLEG): container finished" podID="cccf12d4-0554-4ea8-95f3-ddf4e73317eb" containerID="13fc22bd7cf83bf3b284c2cbfb78b7df10ebc66fec3b06eda29a8352c5925d5b" exitCode=0
Oct 30 13:39:40.305090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.305011    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cccf12d4-0554-4ea8-95f3-ddf4e73317eb-kube-api-access-br2jr\" (UniqueName: \"kubernetes.io/projected/cccf12d4-0554-4ea8-95f3-ddf4e73317eb-kube-api-access-br2jr\") pod \"cccf12d4-0554-4ea8-95f3-ddf4e73317eb\" (UID: \"cccf12d4-0554-4ea8-95f3-ddf4e73317eb\") "
Oct 30 13:39:40.305090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.305050    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cccf12d4-0554-4ea8-95f3-ddf4e73317eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cccf12d4-0554-4ea8-95f3-ddf4e73317eb-plugins-config-volume\") pod \"cccf12d4-0554-4ea8-95f3-ddf4e73317eb\" (UID: \"cccf12d4-0554-4ea8-95f3-ddf4e73317eb\") "
Oct 30 13:39:40.305239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.305147    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n" event={"ID":"cccf12d4-0554-4ea8-95f3-ddf4e73317eb","Type":"ContainerDied","Data":"13fc22bd7cf83bf3b284c2cbfb78b7df10ebc66fec3b06eda29a8352c5925d5b"}
Oct 30 13:39:40.305239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.305191    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n" event={"ID":"cccf12d4-0554-4ea8-95f3-ddf4e73317eb","Type":"ContainerDied","Data":"6f576b3d09cff0a4197249d9c0fa16aa1c5ea1ce30d37ad4596fef3b3ab7fd2d"}
Oct 30 13:39:40.305383 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.305258    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"
Oct 30 13:39:40.306115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.306088    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cccf12d4-0554-4ea8-95f3-ddf4e73317eb-plugins-config-volume" pod "cccf12d4-0554-4ea8-95f3-ddf4e73317eb" (UID: "cccf12d4-0554-4ea8-95f3-ddf4e73317eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:40.308208 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.308181    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cccf12d4-0554-4ea8-95f3-ddf4e73317eb-kube-api-access-br2jr" pod "cccf12d4-0554-4ea8-95f3-ddf4e73317eb" (UID: "cccf12d4-0554-4ea8-95f3-ddf4e73317eb"). InnerVolumeSpecName "kube-api-access-br2jr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:40.316974 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.316949    5893 scope.go:122] "RemoveContainer" containerID="d2d63accbeb2d6150c235f3a0f341989042f24d0ee9729eabb1c03b35193ded7"
Oct 30 13:39:40.317330 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:40.317308    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2d63accbeb2d6150c235f3a0f341989042f24d0ee9729eabb1c03b35193ded7\": container with ID starting with d2d63accbeb2d6150c235f3a0f341989042f24d0ee9729eabb1c03b35193ded7 not found: ID does not exist" containerID="d2d63accbeb2d6150c235f3a0f341989042f24d0ee9729eabb1c03b35193ded7"
Oct 30 13:39:40.317460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.317340    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2d63accbeb2d6150c235f3a0f341989042f24d0ee9729eabb1c03b35193ded7"} err="failed to get container status \"d2d63accbeb2d6150c235f3a0f341989042f24d0ee9729eabb1c03b35193ded7\": rpc error: code = NotFound desc = could not find container \"d2d63accbeb2d6150c235f3a0f341989042f24d0ee9729eabb1c03b35193ded7\": container with ID starting with d2d63accbeb2d6150c235f3a0f341989042f24d0ee9729eabb1c03b35193ded7 not found: ID does not exist"
Oct 30 13:39:40.317460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.317362    5893 scope.go:122] "RemoveContainer" containerID="13fc22bd7cf83bf3b284c2cbfb78b7df10ebc66fec3b06eda29a8352c5925d5b"
Oct 30 13:39:40.341766 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.341741    5893 scope.go:122] "RemoveContainer" containerID="13fc22bd7cf83bf3b284c2cbfb78b7df10ebc66fec3b06eda29a8352c5925d5b"
Oct 30 13:39:40.342137 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:40.342110    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13fc22bd7cf83bf3b284c2cbfb78b7df10ebc66fec3b06eda29a8352c5925d5b\": container with ID starting with 13fc22bd7cf83bf3b284c2cbfb78b7df10ebc66fec3b06eda29a8352c5925d5b not found: ID does not exist" containerID="13fc22bd7cf83bf3b284c2cbfb78b7df10ebc66fec3b06eda29a8352c5925d5b"
Oct 30 13:39:40.342244 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.342147    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13fc22bd7cf83bf3b284c2cbfb78b7df10ebc66fec3b06eda29a8352c5925d5b"} err="failed to get container status \"13fc22bd7cf83bf3b284c2cbfb78b7df10ebc66fec3b06eda29a8352c5925d5b\": rpc error: code = NotFound desc = could not find container \"13fc22bd7cf83bf3b284c2cbfb78b7df10ebc66fec3b06eda29a8352c5925d5b\": container with ID starting with 13fc22bd7cf83bf3b284c2cbfb78b7df10ebc66fec3b06eda29a8352c5925d5b not found: ID does not exist"
Oct 30 13:39:40.406436 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.406352    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/855e2bc2-43d6-4b6a-827f-a778178d6e23-kube-api-access-xj6n7\" (UniqueName: \"kubernetes.io/projected/855e2bc2-43d6-4b6a-827f-a778178d6e23-kube-api-access-xj6n7\") pod \"855e2bc2-43d6-4b6a-827f-a778178d6e23\" (UID: \"855e2bc2-43d6-4b6a-827f-a778178d6e23\") "
Oct 30 13:39:40.406656 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.406636    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/855e2bc2-43d6-4b6a-827f-a778178d6e23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/855e2bc2-43d6-4b6a-827f-a778178d6e23-plugins-config-volume\") pod \"855e2bc2-43d6-4b6a-827f-a778178d6e23\" (UID: \"855e2bc2-43d6-4b6a-827f-a778178d6e23\") "
Oct 30 13:39:40.406965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.406940    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/855e2bc2-43d6-4b6a-827f-a778178d6e23-plugins-config-volume" pod "855e2bc2-43d6-4b6a-827f-a778178d6e23" (UID: "855e2bc2-43d6-4b6a-827f-a778178d6e23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:40.407186 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.407158    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br2jr\" (UniqueName: \"kubernetes.io/projected/cccf12d4-0554-4ea8-95f3-ddf4e73317eb-kube-api-access-br2jr\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:40.407186 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.407180    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cccf12d4-0554-4ea8-95f3-ddf4e73317eb-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:40.407314 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.407195    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/855e2bc2-43d6-4b6a-827f-a778178d6e23-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:40.408778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.408752    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855e2bc2-43d6-4b6a-827f-a778178d6e23-kube-api-access-xj6n7" pod "855e2bc2-43d6-4b6a-827f-a778178d6e23" (UID: "855e2bc2-43d6-4b6a-827f-a778178d6e23"). InnerVolumeSpecName "kube-api-access-xj6n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:40.508057 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.508021    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj6n7\" (UniqueName: \"kubernetes.io/projected/855e2bc2-43d6-4b6a-827f-a778178d6e23-kube-api-access-xj6n7\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:40.641681 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.641640    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"]
Oct 30 13:39:40.646533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.646500    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-0b2c92-646959cb87-n5qj6"]
Oct 30 13:39:40.677610 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.677542    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"]
Oct 30 13:39:40.681182 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:40.681156    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-0b2c92-b489c67b-6c52n"]
Oct 30 13:39:42.531892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:42.531859    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="855e2bc2-43d6-4b6a-827f-a778178d6e23" path="/var/lib/kubelet/pods/855e2bc2-43d6-4b6a-827f-a778178d6e23/volumes"
Oct 30 13:39:42.532394 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:42.532379    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cccf12d4-0554-4ea8-95f3-ddf4e73317eb" path="/var/lib/kubelet/pods/cccf12d4-0554-4ea8-95f3-ddf4e73317eb/volumes"
Oct 30 13:39:44.828412 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:44.828373    5893 patch_prober.go:29] interesting pod/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:39:44.828805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:44.828430    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t" podUID="1f632f65-19d6-4073-bafc-f840bb22ecc7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:39:45.549653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.549588    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:39:45.686916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.686877    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f632f65-19d6-4073-bafc-f840bb22ecc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f632f65-19d6-4073-bafc-f840bb22ecc7-istiod-ca-cert\") pod \"1f632f65-19d6-4073-bafc-f840bb22ecc7\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") "
Oct 30 13:39:45.687094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.687002    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-workload-certs\") pod \"1f632f65-19d6-4073-bafc-f840bb22ecc7\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") "
Oct 30 13:39:45.687094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.687031    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f632f65-19d6-4073-bafc-f840bb22ecc7-kube-api-access-65j8h\" (UniqueName: \"kubernetes.io/projected/1f632f65-19d6-4073-bafc-f840bb22ecc7-kube-api-access-65j8h\") pod \"1f632f65-19d6-4073-bafc-f840bb22ecc7\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") "
Oct 30 13:39:45.687094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.687083    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-podinfo\") pod \"1f632f65-19d6-4073-bafc-f840bb22ecc7\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") "
Oct 30 13:39:45.687377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.687110    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-envoy\") pod \"1f632f65-19d6-4073-bafc-f840bb22ecc7\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") "
Oct 30 13:39:45.687377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.687213    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-workload-socket\") pod \"1f632f65-19d6-4073-bafc-f840bb22ecc7\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") "
Oct 30 13:39:45.687377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.687230    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f632f65-19d6-4073-bafc-f840bb22ecc7-istiod-ca-cert" pod "1f632f65-19d6-4073-bafc-f840bb22ecc7" (UID: "1f632f65-19d6-4073-bafc-f840bb22ecc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:45.687377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.687261    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-token\" (UniqueName: \"kubernetes.io/projected/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-token\") pod \"1f632f65-19d6-4073-bafc-f840bb22ecc7\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") "
Oct 30 13:39:45.687377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.687314    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-data\") pod \"1f632f65-19d6-4073-bafc-f840bb22ecc7\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") "
Oct 30 13:39:45.687377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.687353    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-credential-socket\") pod \"1f632f65-19d6-4073-bafc-f840bb22ecc7\" (UID: \"1f632f65-19d6-4073-bafc-f840bb22ecc7\") "
Oct 30 13:39:45.687699 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.687421    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-workload-certs" pod "1f632f65-19d6-4073-bafc-f840bb22ecc7" (UID: "1f632f65-19d6-4073-bafc-f840bb22ecc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:45.687699 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.687667    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-data" pod "1f632f65-19d6-4073-bafc-f840bb22ecc7" (UID: "1f632f65-19d6-4073-bafc-f840bb22ecc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:45.688204 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.688182    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f632f65-19d6-4073-bafc-f840bb22ecc7-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:45.688204 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.688205    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:45.688424 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.688218    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:45.688424 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.688228    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-credential-socket" pod "1f632f65-19d6-4073-bafc-f840bb22ecc7" (UID: "1f632f65-19d6-4073-bafc-f840bb22ecc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:45.688424 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.688322    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-workload-socket" pod "1f632f65-19d6-4073-bafc-f840bb22ecc7" (UID: "1f632f65-19d6-4073-bafc-f840bb22ecc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:45.690062 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.690036    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-podinfo" pod "1f632f65-19d6-4073-bafc-f840bb22ecc7" (UID: "1f632f65-19d6-4073-bafc-f840bb22ecc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:39:45.690192 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.690072    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-envoy" pod "1f632f65-19d6-4073-bafc-f840bb22ecc7" (UID: "1f632f65-19d6-4073-bafc-f840bb22ecc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:39:45.690192 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.690111    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-token" pod "1f632f65-19d6-4073-bafc-f840bb22ecc7" (UID: "1f632f65-19d6-4073-bafc-f840bb22ecc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:45.690311 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.690185    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f632f65-19d6-4073-bafc-f840bb22ecc7-kube-api-access-65j8h" pod "1f632f65-19d6-4073-bafc-f840bb22ecc7" (UID: "1f632f65-19d6-4073-bafc-f840bb22ecc7"). InnerVolumeSpecName "kube-api-access-65j8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:45.789828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.789795    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65j8h\" (UniqueName: \"kubernetes.io/projected/1f632f65-19d6-4073-bafc-f840bb22ecc7-kube-api-access-65j8h\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:45.789828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.789822    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:45.789828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.789831    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:45.790099 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.789840    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:45.790099 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.789848    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f632f65-19d6-4073-bafc-f840bb22ecc7-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:45.790099 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:45.789856    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f632f65-19d6-4073-bafc-f840bb22ecc7-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:46.350694 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:46.350648    5893 generic.go:356] "Generic (PLEG): container finished" podID="1f632f65-19d6-4073-bafc-f840bb22ecc7" containerID="ab87ac6616b9b43c8df230fb3918f570953cbf1f474783af2aa3c621fcdafe80" exitCode=0
Oct 30 13:39:46.351171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:46.350712    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"
Oct 30 13:39:46.351171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:46.350721    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t" event={"ID":"1f632f65-19d6-4073-bafc-f840bb22ecc7","Type":"ContainerDied","Data":"ab87ac6616b9b43c8df230fb3918f570953cbf1f474783af2aa3c621fcdafe80"}
Oct 30 13:39:46.351171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:46.350824    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t" event={"ID":"1f632f65-19d6-4073-bafc-f840bb22ecc7","Type":"ContainerDied","Data":"1a8e7d539fddafaf3d9bb6dc0502febd3fecf3855081c0cdae27367d74e11a00"}
Oct 30 13:39:46.351171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:46.350854    5893 scope.go:122] "RemoveContainer" containerID="ab87ac6616b9b43c8df230fb3918f570953cbf1f474783af2aa3c621fcdafe80"
Oct 30 13:39:46.371643 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:46.371619    5893 scope.go:122] "RemoveContainer" containerID="ab87ac6616b9b43c8df230fb3918f570953cbf1f474783af2aa3c621fcdafe80"
Oct 30 13:39:46.372046 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:46.372020    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab87ac6616b9b43c8df230fb3918f570953cbf1f474783af2aa3c621fcdafe80\": container with ID starting with ab87ac6616b9b43c8df230fb3918f570953cbf1f474783af2aa3c621fcdafe80 not found: ID does not exist" containerID="ab87ac6616b9b43c8df230fb3918f570953cbf1f474783af2aa3c621fcdafe80"
Oct 30 13:39:46.372163 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:46.372057    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab87ac6616b9b43c8df230fb3918f570953cbf1f474783af2aa3c621fcdafe80"} err="failed to get container status \"ab87ac6616b9b43c8df230fb3918f570953cbf1f474783af2aa3c621fcdafe80\": rpc error: code = NotFound desc = could not find container \"ab87ac6616b9b43c8df230fb3918f570953cbf1f474783af2aa3c621fcdafe80\": container with ID starting with ab87ac6616b9b43c8df230fb3918f570953cbf1f474783af2aa3c621fcdafe80 not found: ID does not exist"
Oct 30 13:39:46.400597 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:46.400568    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"]
Oct 30 13:39:46.410193 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:46.410164    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-e9d641-openshift-default-7ccb47fbd7-5sb9t"]
Oct 30 13:39:46.532770 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:46.532739    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f632f65-19d6-4073-bafc-f840bb22ecc7" path="/var/lib/kubelet/pods/1f632f65-19d6-4073-bafc-f840bb22ecc7/volumes"
Oct 30 13:39:47.318808 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.318768    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch" podUID="262cbe09-e912-4db9-b5ea-3d4ddbd073f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.964685    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"]
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.965864    5893 cpu_manager.go:414] "containerMap: removing container" podUID="98d3dc24-762e-4470-a524-c5d7c822ea80" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.965883    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98d3dc24-762e-4470-a524-c5d7c822ea80" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.965902    5893 cpu_manager.go:414] "containerMap: removing container" podUID="09976b18-da28-4c48-8ac1-c3f914e632b4" containerName="payload-pre-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.965912    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09976b18-da28-4c48-8ac1-c3f914e632b4" containerName="payload-pre-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.965925    5893 cpu_manager.go:414] "containerMap: removing container" podUID="728d41af-901f-4def-a2c1-aee79df80e1e" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.965930    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="728d41af-901f-4def-a2c1-aee79df80e1e" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.965938    5893 cpu_manager.go:414] "containerMap: removing container" podUID="8ac8d101-9ee3-42fa-9719-66220e201923" containerName="maas-api"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.965943    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac8d101-9ee3-42fa-9719-66220e201923" containerName="maas-api"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.965962    5893 cpu_manager.go:414] "containerMap: removing container" podUID="020e8b33-8eb0-4a5e-9de7-28b422ef2b62" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.965967    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="020e8b33-8eb0-4a5e-9de7-28b422ef2b62" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.965973    5893 cpu_manager.go:414] "containerMap: removing container" podUID="bfa3efb6-6744-4a5e-8d30-3c1f23d79b34" containerName="maas-api"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.965979    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa3efb6-6744-4a5e-8d30-3c1f23d79b34" containerName="maas-api"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.965991    5893 cpu_manager.go:414] "containerMap: removing container" podUID="58368e3f-ddb9-4363-9d05-93e5bb0e037e" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.965997    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58368e3f-ddb9-4363-9d05-93e5bb0e037e" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966007    5893 cpu_manager.go:414] "containerMap: removing container" podUID="805ec71c-0b0c-48a9-a154-7fbf568f73ba" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966012    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="805ec71c-0b0c-48a9-a154-7fbf568f73ba" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966018    5893 cpu_manager.go:414] "containerMap: removing container" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966023    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84c68bf3-ee6a-498f-aeb1-0318f788488d" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966033    5893 cpu_manager.go:414] "containerMap: removing container" podUID="21bd8ee2-cb78-4333-bccc-e1c7b34286c7" containerName="payload-pre-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966038    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21bd8ee2-cb78-4333-bccc-e1c7b34286c7" containerName="payload-pre-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966049    5893 cpu_manager.go:414] "containerMap: removing container" podUID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966053    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966073    5893 cpu_manager.go:414] "containerMap: removing container" podUID="530acb52-18b3-4645-aad2-b5d56cc913f5" containerName="payload-pre-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966079    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="530acb52-18b3-4645-aad2-b5d56cc913f5" containerName="payload-pre-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966086    5893 cpu_manager.go:414] "containerMap: removing container" podUID="c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966092    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966100    5893 cpu_manager.go:414] "containerMap: removing container" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966105    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966113    5893 cpu_manager.go:414] "containerMap: removing container" podUID="855e2bc2-43d6-4b6a-827f-a778178d6e23" containerName="payload-pre-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966118    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="855e2bc2-43d6-4b6a-827f-a778178d6e23" containerName="payload-pre-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966127    5893 cpu_manager.go:414] "containerMap: removing container" podUID="f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966133    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966144    5893 cpu_manager.go:414] "containerMap: removing container" podUID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966152    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966158    5893 cpu_manager.go:414] "containerMap: removing container" podUID="467105e6-571f-478b-8324-9e61bc980757" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966166    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="467105e6-571f-478b-8324-9e61bc980757" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966174    5893 cpu_manager.go:414] "containerMap: removing container" podUID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966179    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966189    5893 cpu_manager.go:414] "containerMap: removing container" podUID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966196    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966209    5893 cpu_manager.go:414] "containerMap: removing container" podUID="8525f4ea-caea-4321-8c2e-4c6fa4fe06da" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966217    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8525f4ea-caea-4321-8c2e-4c6fa4fe06da" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966229    5893 cpu_manager.go:414] "containerMap: removing container" podUID="5e678e9e-ec06-4b9e-b1f6-cebb61e59571" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966237    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e678e9e-ec06-4b9e-b1f6-cebb61e59571" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966249    5893 cpu_manager.go:414] "containerMap: removing container" podUID="ddd8433a-82e7-43f7-a036-1c79a64230a1" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966257    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd8433a-82e7-43f7-a036-1c79a64230a1" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966329    5893 cpu_manager.go:414] "containerMap: removing container" podUID="2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966337    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966350    5893 cpu_manager.go:414] "containerMap: removing container" podUID="b75c100c-a950-4e30-9c47-51ec63a8b09a" containerName="maas-api"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966355    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75c100c-a950-4e30-9c47-51ec63a8b09a" containerName="maas-api"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966368    5893 cpu_manager.go:414] "containerMap: removing container" podUID="400e5f9c-8609-4854-af56-2f31915c6f13" containerName="maas-api"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966376    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="400e5f9c-8609-4854-af56-2f31915c6f13" containerName="maas-api"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966388    5893 cpu_manager.go:414] "containerMap: removing container" podUID="1df5b08d-07c1-4075-b8da-9545cabd30d9" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966397    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1df5b08d-07c1-4075-b8da-9545cabd30d9" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966408    5893 cpu_manager.go:414] "containerMap: removing container" podUID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966412    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966423    5893 cpu_manager.go:414] "containerMap: removing container" podUID="1f632f65-19d6-4073-bafc-f840bb22ecc7" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966428    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f632f65-19d6-4073-bafc-f840bb22ecc7" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966441    5893 cpu_manager.go:414] "containerMap: removing container" podUID="54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966446    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966464    5893 cpu_manager.go:414] "containerMap: removing container" podUID="cccf12d4-0554-4ea8-95f3-ddf4e73317eb" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966471    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cccf12d4-0554-4ea8-95f3-ddf4e73317eb" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966483    5893 cpu_manager.go:414] "containerMap: removing container" podUID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966489    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966497    5893 cpu_manager.go:414] "containerMap: removing container" podUID="895b7a55-b271-4b61-a3c5-7411e8b9a9b0" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966502    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="895b7a55-b271-4b61-a3c5-7411e8b9a9b0" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966648    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="728d41af-901f-4def-a2c1-aee79df80e1e" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966660    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac8d101-9ee3-42fa-9719-66220e201923" containerName="maas-api"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966669    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="020e8b33-8eb0-4a5e-9de7-28b422ef2b62" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966678    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa3efb6-6744-4a5e-8d30-3c1f23d79b34" containerName="maas-api"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966686    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="58368e3f-ddb9-4363-9d05-93e5bb0e037e" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966694    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="21bd8ee2-cb78-4333-bccc-e1c7b34286c7" containerName="payload-pre-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966702    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="805ec71c-0b0c-48a9-a154-7fbf568f73ba" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966713    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966723    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="530acb52-18b3-4645-aad2-b5d56cc913f5" containerName="payload-pre-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966735    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7e2c2e2-f1c3-4e05-8366-135e54ad4eaa" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966744    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966750    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="855e2bc2-43d6-4b6a-827f-a778178d6e23" containerName="payload-pre-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966757    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="f79c5dcd-ddd1-44c2-9f5e-93b8e08cc81d" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966762    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966770    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="467105e6-571f-478b-8324-9e61bc980757" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966777    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a767a0-a88a-4a84-b0e1-c918badbd3e9" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966784    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac623bd1-d3de-47f4-9e06-a2261a160df8" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966791    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="8525f4ea-caea-4321-8c2e-4c6fa4fe06da" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966801    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e678e9e-ec06-4b9e-b1f6-cebb61e59571" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966815    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd8433a-82e7-43f7-a036-1c79a64230a1" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966825    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f4edd3c-438c-4c1e-90ea-3ef81b2ad02f" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966833    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75c100c-a950-4e30-9c47-51ec63a8b09a" containerName="maas-api"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966839    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="400e5f9c-8609-4854-af56-2f31915c6f13" containerName="maas-api"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966846    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="1df5b08d-07c1-4075-b8da-9545cabd30d9" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966853    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966859    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f632f65-19d6-4073-bafc-f840bb22ecc7" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966866    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fb1fae-c51f-4f04-a6ae-110f42dfd9eb" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966871    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="cccf12d4-0554-4ea8-95f3-ddf4e73317eb" containerName="payload-processing"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966883    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="721c5b17-8eeb-4e8b-94b4-2d46a18a384c" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966894    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="895b7a55-b271-4b61-a3c5-7411e8b9a9b0" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966904    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="98d3dc24-762e-4470-a524-c5d7c822ea80" containerName="istio-proxy"
Oct 30 13:39:47.971327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.966914    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="09976b18-da28-4c48-8ac1-c3f914e632b4" containerName="payload-pre-processing"
Oct 30 13:39:47.978488 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.972555    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"
Oct 30 13:39:47.997326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.990791    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"]
Oct 30 13:39:47.997326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.993256    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ba8145\""
Oct 30 13:39:47.997326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:47.993421    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ba8145-dockercfg-gs9nb\""
Oct 30 13:39:48.030010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.029980    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvmk\" (UniqueName: \"kubernetes.io/projected/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3-kube-api-access-hwvmk\") pod \"payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2\" (UID: \"f02e2238-4a09-4cf3-b20a-d82b0aba4fe3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"
Oct 30 13:39:48.030229 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.030039    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2\" (UID: \"f02e2238-4a09-4cf3-b20a-d82b0aba4fe3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"
Oct 30 13:39:48.131709 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.131643    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvmk\" (UniqueName: \"kubernetes.io/projected/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3-kube-api-access-hwvmk\") pod \"payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2\" (UID: \"f02e2238-4a09-4cf3-b20a-d82b0aba4fe3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"
Oct 30 13:39:48.131953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.131804    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2\" (UID: \"f02e2238-4a09-4cf3-b20a-d82b0aba4fe3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"
Oct 30 13:39:48.132633 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.132610    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2\" (UID: \"f02e2238-4a09-4cf3-b20a-d82b0aba4fe3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"
Oct 30 13:39:48.137284 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.137246    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"]
Oct 30 13:39:48.137878 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.137863    5893 cpu_manager.go:414] "containerMap: removing container" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerName="istio-proxy"
Oct 30 13:39:48.137928 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.137880    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerName="istio-proxy"
Oct 30 13:39:48.138006 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.137996    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1530d7-d75e-4d05-9fda-46f3a07b60c1" containerName="istio-proxy"
Oct 30 13:39:48.141457 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.141444    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"
Oct 30 13:39:48.158498 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.158478    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvmk\" (UniqueName: \"kubernetes.io/projected/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3-kube-api-access-hwvmk\") pod \"payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2\" (UID: \"f02e2238-4a09-4cf3-b20a-d82b0aba4fe3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"
Oct 30 13:39:48.167432 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.167407    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"]
Oct 30 13:39:48.233252 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.233159    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z95nm\" (UniqueName: \"kubernetes.io/projected/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9-kube-api-access-z95nm\") pod \"payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786\" (UID: \"fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"
Oct 30 13:39:48.233475 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.233451    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786\" (UID: \"fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"
Oct 30 13:39:48.294710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.294674    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"
Oct 30 13:39:48.334452 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.334423    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786\" (UID: \"fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"
Oct 30 13:39:48.334651 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.334553    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z95nm\" (UniqueName: \"kubernetes.io/projected/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9-kube-api-access-z95nm\") pod \"payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786\" (UID: \"fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"
Oct 30 13:39:48.335219 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.335196    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786\" (UID: \"fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"
Oct 30 13:39:48.358174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.358148    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95nm\" (UniqueName: \"kubernetes.io/projected/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9-kube-api-access-z95nm\") pod \"payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786\" (UID: \"fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"
Oct 30 13:39:48.427630 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.427264    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"]
Oct 30 13:39:48.427630 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.427537    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c" podUID="6311bfd1-218e-4fe6-a79c-74f3d127e805" containerName="maas-api" containerID="cri-o://3a8fcbd6158c96b6a981624a83de24395ab08d27dc90da343f299fa766b6101d" gracePeriod=30
Oct 30 13:39:48.455408 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.455378    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"
Oct 30 13:39:48.616619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.616587    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"]
Oct 30 13:39:48.626200 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:39:48.626121    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf02e2238_4a09_4cf3_b20a_d82b0aba4fe3.slice/crio-b0a94e089ee51230deb62f57d1a77b05b022e6f6c73ec94e2bd03b9f28337b84 WatchSource:0}: Error finding container b0a94e089ee51230deb62f57d1a77b05b022e6f6c73ec94e2bd03b9f28337b84: Status 404 returned error can't find the container with id b0a94e089ee51230deb62f57d1a77b05b022e6f6c73ec94e2bd03b9f28337b84
Oct 30 13:39:48.664884 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:48.664757    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:39:48.745336 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.745313    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:39:48.778825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.778736    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"]
Oct 30 13:39:48.847339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.843910    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6311bfd1-218e-4fe6-a79c-74f3d127e805-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6311bfd1-218e-4fe6-a79c-74f3d127e805-metrics-certs\") pod \"6311bfd1-218e-4fe6-a79c-74f3d127e805\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") "
Oct 30 13:39:48.847339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.844042    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6311bfd1-218e-4fe6-a79c-74f3d127e805-kube-api-access-h69g5\" (UniqueName: \"kubernetes.io/projected/6311bfd1-218e-4fe6-a79c-74f3d127e805-kube-api-access-h69g5\") pod \"6311bfd1-218e-4fe6-a79c-74f3d127e805\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") "
Oct 30 13:39:48.847339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.844198    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6311bfd1-218e-4fe6-a79c-74f3d127e805-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6311bfd1-218e-4fe6-a79c-74f3d127e805-maas-api-tls\") pod \"6311bfd1-218e-4fe6-a79c-74f3d127e805\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") "
Oct 30 13:39:48.847339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.844308    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6311bfd1-218e-4fe6-a79c-74f3d127e805-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6311bfd1-218e-4fe6-a79c-74f3d127e805-openshift-service-ca\") pod \"6311bfd1-218e-4fe6-a79c-74f3d127e805\" (UID: \"6311bfd1-218e-4fe6-a79c-74f3d127e805\") "
Oct 30 13:39:48.847339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.845160    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6311bfd1-218e-4fe6-a79c-74f3d127e805-openshift-service-ca" pod "6311bfd1-218e-4fe6-a79c-74f3d127e805" (UID: "6311bfd1-218e-4fe6-a79c-74f3d127e805"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:48.847339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.847253    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6311bfd1-218e-4fe6-a79c-74f3d127e805-metrics-certs" pod "6311bfd1-218e-4fe6-a79c-74f3d127e805" (UID: "6311bfd1-218e-4fe6-a79c-74f3d127e805"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:48.847797 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.847508    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6311bfd1-218e-4fe6-a79c-74f3d127e805-kube-api-access-h69g5" pod "6311bfd1-218e-4fe6-a79c-74f3d127e805" (UID: "6311bfd1-218e-4fe6-a79c-74f3d127e805"). InnerVolumeSpecName "kube-api-access-h69g5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:48.849582 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.849551    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6311bfd1-218e-4fe6-a79c-74f3d127e805-maas-api-tls" pod "6311bfd1-218e-4fe6-a79c-74f3d127e805" (UID: "6311bfd1-218e-4fe6-a79c-74f3d127e805"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:48.878303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.878253    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"]
Oct 30 13:39:48.878551 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.878505    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5" podUID="00423102-32b5-4a2a-8f6d-e93309aae3b6" containerName="maas-api" containerID="cri-o://545a93a7e800b4f316819642d23eab81192e2a5fe70ba70e127e3d728938a8ff" gracePeriod=30
Oct 30 13:39:48.949459 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.949363    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6311bfd1-218e-4fe6-a79c-74f3d127e805-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:48.949459 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.949401    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6311bfd1-218e-4fe6-a79c-74f3d127e805-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:48.949459 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.949416    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h69g5\" (UniqueName: \"kubernetes.io/projected/6311bfd1-218e-4fe6-a79c-74f3d127e805-kube-api-access-h69g5\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:48.949459 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:48.949430    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6311bfd1-218e-4fe6-a79c-74f3d127e805-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:49.156659 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.156632    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:39:49.254428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.254395    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00423102-32b5-4a2a-8f6d-e93309aae3b6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00423102-32b5-4a2a-8f6d-e93309aae3b6-openshift-service-ca\") pod \"00423102-32b5-4a2a-8f6d-e93309aae3b6\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") "
Oct 30 13:39:49.254627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.254497    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00423102-32b5-4a2a-8f6d-e93309aae3b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00423102-32b5-4a2a-8f6d-e93309aae3b6-metrics-certs\") pod \"00423102-32b5-4a2a-8f6d-e93309aae3b6\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") "
Oct 30 13:39:49.254627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.254537    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00423102-32b5-4a2a-8f6d-e93309aae3b6-kube-api-access-4zbhr\" (UniqueName: \"kubernetes.io/projected/00423102-32b5-4a2a-8f6d-e93309aae3b6-kube-api-access-4zbhr\") pod \"00423102-32b5-4a2a-8f6d-e93309aae3b6\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") "
Oct 30 13:39:49.255396 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.254626    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00423102-32b5-4a2a-8f6d-e93309aae3b6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00423102-32b5-4a2a-8f6d-e93309aae3b6-maas-api-tls\") pod \"00423102-32b5-4a2a-8f6d-e93309aae3b6\" (UID: \"00423102-32b5-4a2a-8f6d-e93309aae3b6\") "
Oct 30 13:39:49.255396 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.254846    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00423102-32b5-4a2a-8f6d-e93309aae3b6-openshift-service-ca" pod "00423102-32b5-4a2a-8f6d-e93309aae3b6" (UID: "00423102-32b5-4a2a-8f6d-e93309aae3b6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:49.255750 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.255633    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00423102-32b5-4a2a-8f6d-e93309aae3b6-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:49.257020 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.256988    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00423102-32b5-4a2a-8f6d-e93309aae3b6-kube-api-access-4zbhr" pod "00423102-32b5-4a2a-8f6d-e93309aae3b6" (UID: "00423102-32b5-4a2a-8f6d-e93309aae3b6"). InnerVolumeSpecName "kube-api-access-4zbhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:49.257186 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.257162    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00423102-32b5-4a2a-8f6d-e93309aae3b6-maas-api-tls" pod "00423102-32b5-4a2a-8f6d-e93309aae3b6" (UID: "00423102-32b5-4a2a-8f6d-e93309aae3b6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:49.257468 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.257439    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00423102-32b5-4a2a-8f6d-e93309aae3b6-metrics-certs" pod "00423102-32b5-4a2a-8f6d-e93309aae3b6" (UID: "00423102-32b5-4a2a-8f6d-e93309aae3b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:49.358308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.358122    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00423102-32b5-4a2a-8f6d-e93309aae3b6-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:49.358308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.358161    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zbhr\" (UniqueName: \"kubernetes.io/projected/00423102-32b5-4a2a-8f6d-e93309aae3b6-kube-api-access-4zbhr\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:49.358308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.358176    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00423102-32b5-4a2a-8f6d-e93309aae3b6-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:49.390064 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.390026    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786" event={"ID":"fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9","Type":"ContainerStarted","Data":"34c72003bd00d222f337d3e1af39ab62b31b5d2e662a2f739dbd3aba43f17ec2"}
Oct 30 13:39:49.390372 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.390350    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786" event={"ID":"fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9","Type":"ContainerStarted","Data":"ab7316e46ff82edf6632645ced5f61781b26076cf4b0038e506183c406401e3b"}
Oct 30 13:39:49.391486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.391466    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"
Oct 30 13:39:49.393623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.393597    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2" event={"ID":"f02e2238-4a09-4cf3-b20a-d82b0aba4fe3","Type":"ContainerStarted","Data":"ab20139048eb3705ef95be13b022f4fe71e741b9885e8b061f84532408549e2f"}
Oct 30 13:39:49.394988 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.394970    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2" event={"ID":"f02e2238-4a09-4cf3-b20a-d82b0aba4fe3","Type":"ContainerStarted","Data":"b0a94e089ee51230deb62f57d1a77b05b022e6f6c73ec94e2bd03b9f28337b84"}
Oct 30 13:39:49.395122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.395102    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"
Oct 30 13:39:49.399682 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.399657    5893 generic.go:356] "Generic (PLEG): container finished" podID="00423102-32b5-4a2a-8f6d-e93309aae3b6" containerID="545a93a7e800b4f316819642d23eab81192e2a5fe70ba70e127e3d728938a8ff" exitCode=0
Oct 30 13:39:49.400309 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.399782    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5" event={"ID":"00423102-32b5-4a2a-8f6d-e93309aae3b6","Type":"ContainerDied","Data":"545a93a7e800b4f316819642d23eab81192e2a5fe70ba70e127e3d728938a8ff"}
Oct 30 13:39:49.400309 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.399825    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5" event={"ID":"00423102-32b5-4a2a-8f6d-e93309aae3b6","Type":"ContainerDied","Data":"e4dfe804d014738d619e155e440fd1825af55b8e1dd54d1a3393f0a93625dd13"}
Oct 30 13:39:49.400309 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.399856    5893 scope.go:122] "RemoveContainer" containerID="545a93a7e800b4f316819642d23eab81192e2a5fe70ba70e127e3d728938a8ff"
Oct 30 13:39:49.400309 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.399954    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"
Oct 30 13:39:49.408404 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.408377    5893 generic.go:356] "Generic (PLEG): container finished" podID="6311bfd1-218e-4fe6-a79c-74f3d127e805" containerID="3a8fcbd6158c96b6a981624a83de24395ab08d27dc90da343f299fa766b6101d" exitCode=0
Oct 30 13:39:49.408559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.408429    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"
Oct 30 13:39:49.408559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.408484    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c" event={"ID":"6311bfd1-218e-4fe6-a79c-74f3d127e805","Type":"ContainerDied","Data":"3a8fcbd6158c96b6a981624a83de24395ab08d27dc90da343f299fa766b6101d"}
Oct 30 13:39:49.408559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.408525    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c" event={"ID":"6311bfd1-218e-4fe6-a79c-74f3d127e805","Type":"ContainerDied","Data":"36453df53695b065efed2ad44044bec250ab69ec213d92b73e720ad30c4d01ac"}
Oct 30 13:39:49.420059 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.420038    5893 scope.go:122] "RemoveContainer" containerID="545a93a7e800b4f316819642d23eab81192e2a5fe70ba70e127e3d728938a8ff"
Oct 30 13:39:49.420431 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:49.420405    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545a93a7e800b4f316819642d23eab81192e2a5fe70ba70e127e3d728938a8ff\": container with ID starting with 545a93a7e800b4f316819642d23eab81192e2a5fe70ba70e127e3d728938a8ff not found: ID does not exist" containerID="545a93a7e800b4f316819642d23eab81192e2a5fe70ba70e127e3d728938a8ff"
Oct 30 13:39:49.420497 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.420441    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545a93a7e800b4f316819642d23eab81192e2a5fe70ba70e127e3d728938a8ff"} err="failed to get container status \"545a93a7e800b4f316819642d23eab81192e2a5fe70ba70e127e3d728938a8ff\": rpc error: code = NotFound desc = could not find container \"545a93a7e800b4f316819642d23eab81192e2a5fe70ba70e127e3d728938a8ff\": container with ID starting with 545a93a7e800b4f316819642d23eab81192e2a5fe70ba70e127e3d728938a8ff not found: ID does not exist"
Oct 30 13:39:49.420497 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.420463    5893 scope.go:122] "RemoveContainer" containerID="3a8fcbd6158c96b6a981624a83de24395ab08d27dc90da343f299fa766b6101d"
Oct 30 13:39:49.430350 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.430331    5893 scope.go:122] "RemoveContainer" containerID="3a8fcbd6158c96b6a981624a83de24395ab08d27dc90da343f299fa766b6101d"
Oct 30 13:39:49.430598 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:49.430576    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8fcbd6158c96b6a981624a83de24395ab08d27dc90da343f299fa766b6101d\": container with ID starting with 3a8fcbd6158c96b6a981624a83de24395ab08d27dc90da343f299fa766b6101d not found: ID does not exist" containerID="3a8fcbd6158c96b6a981624a83de24395ab08d27dc90da343f299fa766b6101d"
Oct 30 13:39:49.430649 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.430606    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8fcbd6158c96b6a981624a83de24395ab08d27dc90da343f299fa766b6101d"} err="failed to get container status \"3a8fcbd6158c96b6a981624a83de24395ab08d27dc90da343f299fa766b6101d\": rpc error: code = NotFound desc = could not find container \"3a8fcbd6158c96b6a981624a83de24395ab08d27dc90da343f299fa766b6101d\": container with ID starting with 3a8fcbd6158c96b6a981624a83de24395ab08d27dc90da343f299fa766b6101d not found: ID does not exist"
Oct 30 13:39:49.441362 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.436783    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786" podStartSLOduration=1.436763512 podStartE2EDuration="1.436763512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:49.432949109 +0000 UTC m=+3231.016378819" watchObservedRunningTime="2026-10-30 13:39:49.436763512 +0000 UTC m=+3231.020193205"
Oct 30 13:39:49.466892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.466850    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"]
Oct 30 13:39:49.471596 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.471570    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-0b2c92-749b486c97-qtz7c"]
Oct 30 13:39:49.544514 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.544458    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2" podStartSLOduration=2.5444408530000002 podStartE2EDuration="2.544440853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:49.507876915 +0000 UTC m=+3231.091306608" watchObservedRunningTime="2026-10-30 13:39:49.544440853 +0000 UTC m=+3231.127870547"
Oct 30 13:39:49.544805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.544632    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"]
Oct 30 13:39:49.549054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:49.549023    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-e9d641-6bbbb7674f-kdxc5"]
Oct 30 13:39:50.532827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:50.532785    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00423102-32b5-4a2a-8f6d-e93309aae3b6" path="/var/lib/kubelet/pods/00423102-32b5-4a2a-8f6d-e93309aae3b6/volumes"
Oct 30 13:39:50.533498 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:50.533478    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6311bfd1-218e-4fe6-a79c-74f3d127e805" path="/var/lib/kubelet/pods/6311bfd1-218e-4fe6-a79c-74f3d127e805/volumes"
Oct 30 13:39:55.714599 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:55.714566    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"]
Oct 30 13:39:55.715129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:55.714833    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64" podUID="25b93077-be04-4210-9cd9-c4b1f182a1d1" containerName="maas-api" containerID="cri-o://b7be9ed899d2d72567f3aa3dd5db096ccae666765f159b9998eb86b289d19e25" gracePeriod=30
Oct 30 13:39:55.997473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:55.997449    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:39:56.148115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.148074    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b93077-be04-4210-9cd9-c4b1f182a1d1-kube-api-access-bbq49\" (UniqueName: \"kubernetes.io/projected/25b93077-be04-4210-9cd9-c4b1f182a1d1-kube-api-access-bbq49\") pod \"25b93077-be04-4210-9cd9-c4b1f182a1d1\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") "
Oct 30 13:39:56.148380 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.148245    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25b93077-be04-4210-9cd9-c4b1f182a1d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25b93077-be04-4210-9cd9-c4b1f182a1d1-maas-api-tls\") pod \"25b93077-be04-4210-9cd9-c4b1f182a1d1\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") "
Oct 30 13:39:56.148380 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.148372    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25b93077-be04-4210-9cd9-c4b1f182a1d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b93077-be04-4210-9cd9-c4b1f182a1d1-metrics-certs\") pod \"25b93077-be04-4210-9cd9-c4b1f182a1d1\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") "
Oct 30 13:39:56.148506 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.148397    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b93077-be04-4210-9cd9-c4b1f182a1d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b93077-be04-4210-9cd9-c4b1f182a1d1-openshift-service-ca\") pod \"25b93077-be04-4210-9cd9-c4b1f182a1d1\" (UID: \"25b93077-be04-4210-9cd9-c4b1f182a1d1\") "
Oct 30 13:39:56.148895 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.148862    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b93077-be04-4210-9cd9-c4b1f182a1d1-openshift-service-ca" pod "25b93077-be04-4210-9cd9-c4b1f182a1d1" (UID: "25b93077-be04-4210-9cd9-c4b1f182a1d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:56.151090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.151057    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b93077-be04-4210-9cd9-c4b1f182a1d1-metrics-certs" pod "25b93077-be04-4210-9cd9-c4b1f182a1d1" (UID: "25b93077-be04-4210-9cd9-c4b1f182a1d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:56.151300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.151144    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b93077-be04-4210-9cd9-c4b1f182a1d1-maas-api-tls" pod "25b93077-be04-4210-9cd9-c4b1f182a1d1" (UID: "25b93077-be04-4210-9cd9-c4b1f182a1d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:39:56.151300 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.151172    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b93077-be04-4210-9cd9-c4b1f182a1d1-kube-api-access-bbq49" pod "25b93077-be04-4210-9cd9-c4b1f182a1d1" (UID: "25b93077-be04-4210-9cd9-c4b1f182a1d1"). InnerVolumeSpecName "kube-api-access-bbq49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:56.291369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.291258    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbq49\" (UniqueName: \"kubernetes.io/projected/25b93077-be04-4210-9cd9-c4b1f182a1d1-kube-api-access-bbq49\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:56.291576 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.291560    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25b93077-be04-4210-9cd9-c4b1f182a1d1-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:56.291679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.291668    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25b93077-be04-4210-9cd9-c4b1f182a1d1-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:56.291787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.291762    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25b93077-be04-4210-9cd9-c4b1f182a1d1-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:56.459990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.459952    5893 generic.go:356] "Generic (PLEG): container finished" podID="25b93077-be04-4210-9cd9-c4b1f182a1d1" containerID="b7be9ed899d2d72567f3aa3dd5db096ccae666765f159b9998eb86b289d19e25" exitCode=0
Oct 30 13:39:56.460151 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.460005    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"
Oct 30 13:39:56.460151 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.460043    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64" event={"ID":"25b93077-be04-4210-9cd9-c4b1f182a1d1","Type":"ContainerDied","Data":"b7be9ed899d2d72567f3aa3dd5db096ccae666765f159b9998eb86b289d19e25"}
Oct 30 13:39:56.460151 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.460108    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64" event={"ID":"25b93077-be04-4210-9cd9-c4b1f182a1d1","Type":"ContainerDied","Data":"84cd1eef75e62a71efe9d2f23057cf0ddd69546a0828e9b4d550a47c2680d46f"}
Oct 30 13:39:56.460151 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.460139    5893 scope.go:122] "RemoveContainer" containerID="b7be9ed899d2d72567f3aa3dd5db096ccae666765f159b9998eb86b289d19e25"
Oct 30 13:39:56.474304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.474251    5893 scope.go:122] "RemoveContainer" containerID="b7be9ed899d2d72567f3aa3dd5db096ccae666765f159b9998eb86b289d19e25"
Oct 30 13:39:56.478496 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:56.478464    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7be9ed899d2d72567f3aa3dd5db096ccae666765f159b9998eb86b289d19e25\": container with ID starting with b7be9ed899d2d72567f3aa3dd5db096ccae666765f159b9998eb86b289d19e25 not found: ID does not exist" containerID="b7be9ed899d2d72567f3aa3dd5db096ccae666765f159b9998eb86b289d19e25"
Oct 30 13:39:56.478621 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.478507    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7be9ed899d2d72567f3aa3dd5db096ccae666765f159b9998eb86b289d19e25"} err="failed to get container status \"b7be9ed899d2d72567f3aa3dd5db096ccae666765f159b9998eb86b289d19e25\": rpc error: code = NotFound desc = could not find container \"b7be9ed899d2d72567f3aa3dd5db096ccae666765f159b9998eb86b289d19e25\": container with ID starting with b7be9ed899d2d72567f3aa3dd5db096ccae666765f159b9998eb86b289d19e25 not found: ID does not exist"
Oct 30 13:39:56.505912 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.505888    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"]
Oct 30 13:39:56.511663 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.511636    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-8a5d5e-6cf77f94bd-p8m64"]
Oct 30 13:39:56.532062 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:56.532037    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b93077-be04-4210-9cd9-c4b1f182a1d1" path="/var/lib/kubelet/pods/25b93077-be04-4210-9cd9-c4b1f182a1d1/volumes"
Oct 30 13:39:57.319506 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:57.319467    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch" podUID="262cbe09-e912-4db9-b5ea-3d4ddbd073f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:39:59.411391 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.411364    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"
Oct 30 13:39:59.430807 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.430780    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/262cbe09-e912-4db9-b5ea-3d4ddbd073f5-kube-api-access-2d8hx\" (UniqueName: \"kubernetes.io/projected/262cbe09-e912-4db9-b5ea-3d4ddbd073f5-kube-api-access-2d8hx\") pod \"262cbe09-e912-4db9-b5ea-3d4ddbd073f5\" (UID: \"262cbe09-e912-4db9-b5ea-3d4ddbd073f5\") "
Oct 30 13:39:59.430943 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.430911    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/262cbe09-e912-4db9-b5ea-3d4ddbd073f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262cbe09-e912-4db9-b5ea-3d4ddbd073f5-plugins-config-volume\") pod \"262cbe09-e912-4db9-b5ea-3d4ddbd073f5\" (UID: \"262cbe09-e912-4db9-b5ea-3d4ddbd073f5\") "
Oct 30 13:39:59.431363 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.431337    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262cbe09-e912-4db9-b5ea-3d4ddbd073f5-plugins-config-volume" pod "262cbe09-e912-4db9-b5ea-3d4ddbd073f5" (UID: "262cbe09-e912-4db9-b5ea-3d4ddbd073f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:59.431508 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.431495    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/262cbe09-e912-4db9-b5ea-3d4ddbd073f5-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:59.433094 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.433075    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262cbe09-e912-4db9-b5ea-3d4ddbd073f5-kube-api-access-2d8hx" pod "262cbe09-e912-4db9-b5ea-3d4ddbd073f5" (UID: "262cbe09-e912-4db9-b5ea-3d4ddbd073f5"). InnerVolumeSpecName "kube-api-access-2d8hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:59.494010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.493969    5893 generic.go:356] "Generic (PLEG): container finished" podID="262cbe09-e912-4db9-b5ea-3d4ddbd073f5" containerID="cd199e634d6ac0a0db6ea45adec319b323e7cb36cdde3ccea9969cee1429041a" exitCode=0
Oct 30 13:39:59.494210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.494105    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch" event={"ID":"262cbe09-e912-4db9-b5ea-3d4ddbd073f5","Type":"ContainerDied","Data":"cd199e634d6ac0a0db6ea45adec319b323e7cb36cdde3ccea9969cee1429041a"}
Oct 30 13:39:59.494296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.494205    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch" event={"ID":"262cbe09-e912-4db9-b5ea-3d4ddbd073f5","Type":"ContainerDied","Data":"4b70adfad3139c1c7f5c12fa64a8ebd7201586f363b7719decd01f7afe2b8e7e"}
Oct 30 13:39:59.494296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.494247    5893 scope.go:122] "RemoveContainer" containerID="cd199e634d6ac0a0db6ea45adec319b323e7cb36cdde3ccea9969cee1429041a"
Oct 30 13:39:59.494472 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.494455    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"
Oct 30 13:39:59.511616 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.511597    5893 scope.go:122] "RemoveContainer" containerID="cd199e634d6ac0a0db6ea45adec319b323e7cb36cdde3ccea9969cee1429041a"
Oct 30 13:39:59.511938 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:39:59.511917    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd199e634d6ac0a0db6ea45adec319b323e7cb36cdde3ccea9969cee1429041a\": container with ID starting with cd199e634d6ac0a0db6ea45adec319b323e7cb36cdde3ccea9969cee1429041a not found: ID does not exist" containerID="cd199e634d6ac0a0db6ea45adec319b323e7cb36cdde3ccea9969cee1429041a"
Oct 30 13:39:59.511987 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.511948    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd199e634d6ac0a0db6ea45adec319b323e7cb36cdde3ccea9969cee1429041a"} err="failed to get container status \"cd199e634d6ac0a0db6ea45adec319b323e7cb36cdde3ccea9969cee1429041a\": rpc error: code = NotFound desc = could not find container \"cd199e634d6ac0a0db6ea45adec319b323e7cb36cdde3ccea9969cee1429041a\": container with ID starting with cd199e634d6ac0a0db6ea45adec319b323e7cb36cdde3ccea9969cee1429041a not found: ID does not exist"
Oct 30 13:39:59.532840 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.532812    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d8hx\" (UniqueName: \"kubernetes.io/projected/262cbe09-e912-4db9-b5ea-3d4ddbd073f5-kube-api-access-2d8hx\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:39:59.545395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.545371    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"]
Oct 30 13:39:59.550848 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:39:59.550828    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-7a71ca-5df95d5f8c-tk7ch"]
Oct 30 13:40:00.420714 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:00.420681    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"
Oct 30 13:40:00.533857 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:00.533825    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="262cbe09-e912-4db9-b5ea-3d4ddbd073f5" path="/var/lib/kubelet/pods/262cbe09-e912-4db9-b5ea-3d4ddbd073f5/volumes"
Oct 30 13:40:01.427054 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:01.427022    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"
Oct 30 13:40:03.304630 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:03.304598    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"]
Oct 30 13:40:03.305148 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:03.304828    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85" podUID="abf058d4-f482-47b8-8610-6491dfd318c2" containerName="istio-proxy" containerID="cri-o://3914cf6f21d2753ec2dd310233301b49d94968f5b70859534538a7465d1aa878" gracePeriod=30
Oct 30 13:40:04.652675 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:04.652643    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"]
Oct 30 13:40:08.576921 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.576803    5893 generic.go:356] "Generic (PLEG): container finished" podID="abf058d4-f482-47b8-8610-6491dfd318c2" containerID="3914cf6f21d2753ec2dd310233301b49d94968f5b70859534538a7465d1aa878" exitCode=0
Oct 30 13:40:08.577458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.576854    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85" event={"ID":"abf058d4-f482-47b8-8610-6491dfd318c2","Type":"ContainerDied","Data":"3914cf6f21d2753ec2dd310233301b49d94968f5b70859534538a7465d1aa878"}
Oct 30 13:40:08.577558 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.577512    5893 scope.go:122] "RemoveContainer" containerID="eaadaa18fa1d64fd72ea8d05ffb70b378363f5e2dfe40038e30dce43fe16ca6a"
Oct 30 13:40:08.620225 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.620197    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:40:08.747076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.747036    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"]
Oct 30 13:40:08.766202 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.766170    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf058d4-f482-47b8-8610-6491dfd318c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf058d4-f482-47b8-8610-6491dfd318c2-istiod-ca-cert\") pod \"abf058d4-f482-47b8-8610-6491dfd318c2\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") "
Oct 30 13:40:08.766433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.766245    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-istio-data\") pod \"abf058d4-f482-47b8-8610-6491dfd318c2\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") "
Oct 30 13:40:08.766433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.766292    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-istio-envoy\") pod \"abf058d4-f482-47b8-8610-6491dfd318c2\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") "
Oct 30 13:40:08.766433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.766385    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf058d4-f482-47b8-8610-6491dfd318c2-kube-api-access-rrwnl\" (UniqueName: \"kubernetes.io/projected/abf058d4-f482-47b8-8610-6491dfd318c2-kube-api-access-rrwnl\") pod \"abf058d4-f482-47b8-8610-6491dfd318c2\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") "
Oct 30 13:40:08.766433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.766405    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf058d4-f482-47b8-8610-6491dfd318c2-istio-token\" (UniqueName: \"kubernetes.io/projected/abf058d4-f482-47b8-8610-6491dfd318c2-istio-token\") pod \"abf058d4-f482-47b8-8610-6491dfd318c2\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") "
Oct 30 13:40:08.766690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.766462    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-workload-socket\") pod \"abf058d4-f482-47b8-8610-6491dfd318c2\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") "
Oct 30 13:40:08.766690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.766510    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abf058d4-f482-47b8-8610-6491dfd318c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf058d4-f482-47b8-8610-6491dfd318c2-istio-podinfo\") pod \"abf058d4-f482-47b8-8610-6491dfd318c2\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") "
Oct 30 13:40:08.766690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.766553    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-workload-certs\") pod \"abf058d4-f482-47b8-8610-6491dfd318c2\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") "
Oct 30 13:40:08.766690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.766586    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-credential-socket\") pod \"abf058d4-f482-47b8-8610-6491dfd318c2\" (UID: \"abf058d4-f482-47b8-8610-6491dfd318c2\") "
Oct 30 13:40:08.766690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.766620    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-istio-data" pod "abf058d4-f482-47b8-8610-6491dfd318c2" (UID: "abf058d4-f482-47b8-8610-6491dfd318c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:08.766690 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.766641    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf058d4-f482-47b8-8610-6491dfd318c2-istiod-ca-cert" pod "abf058d4-f482-47b8-8610-6491dfd318c2" (UID: "abf058d4-f482-47b8-8610-6491dfd318c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:08.767125 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.767005    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:08.767125 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.767024    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abf058d4-f482-47b8-8610-6491dfd318c2-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:08.767381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.767158    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-workload-socket" pod "abf058d4-f482-47b8-8610-6491dfd318c2" (UID: "abf058d4-f482-47b8-8610-6491dfd318c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:08.767381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.767244    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-credential-socket" pod "abf058d4-f482-47b8-8610-6491dfd318c2" (UID: "abf058d4-f482-47b8-8610-6491dfd318c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:08.767566 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.767481    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-workload-certs" pod "abf058d4-f482-47b8-8610-6491dfd318c2" (UID: "abf058d4-f482-47b8-8610-6491dfd318c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:08.770264 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.770142    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abf058d4-f482-47b8-8610-6491dfd318c2-istio-podinfo" pod "abf058d4-f482-47b8-8610-6491dfd318c2" (UID: "abf058d4-f482-47b8-8610-6491dfd318c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:40:08.770788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.770756    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf058d4-f482-47b8-8610-6491dfd318c2-kube-api-access-rrwnl" pod "abf058d4-f482-47b8-8610-6491dfd318c2" (UID: "abf058d4-f482-47b8-8610-6491dfd318c2"). InnerVolumeSpecName "kube-api-access-rrwnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:08.771009 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.770991    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-istio-envoy" pod "abf058d4-f482-47b8-8610-6491dfd318c2" (UID: "abf058d4-f482-47b8-8610-6491dfd318c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:08.771256 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.771240    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf058d4-f482-47b8-8610-6491dfd318c2-istio-token" pod "abf058d4-f482-47b8-8610-6491dfd318c2" (UID: "abf058d4-f482-47b8-8610-6491dfd318c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:08.867933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.867850    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:08.867933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.867883    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrwnl\" (UniqueName: \"kubernetes.io/projected/abf058d4-f482-47b8-8610-6491dfd318c2-kube-api-access-rrwnl\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:08.867933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.867894    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abf058d4-f482-47b8-8610-6491dfd318c2-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:08.867933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.867915    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:08.867933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.867926    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abf058d4-f482-47b8-8610-6491dfd318c2-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:08.867933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.867935    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:08.868436 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:08.867945    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abf058d4-f482-47b8-8610-6491dfd318c2-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:09.108639 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:09.108603    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"]
Oct 30 13:40:09.146760 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:09.146673    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"]
Oct 30 13:40:09.587710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:09.587684    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"
Oct 30 13:40:09.587710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:09.587698    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85" event={"ID":"abf058d4-f482-47b8-8610-6491dfd318c2","Type":"ContainerDied","Data":"67a3ebc520c9e7e1b740c3a9a71f4e4b57622d0e154be9b0a7c3964ffa1d726d"}
Oct 30 13:40:09.588318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:09.587767    5893 scope.go:122] "RemoveContainer" containerID="3914cf6f21d2753ec2dd310233301b49d94968f5b70859534538a7465d1aa878"
Oct 30 13:40:09.645705 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:09.645670    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"]
Oct 30 13:40:09.651611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:09.651573    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a71ca-openshift-default-67b645d9dc-d9h85"]
Oct 30 13:40:10.531327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:10.531286    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf058d4-f482-47b8-8610-6491dfd318c2" path="/var/lib/kubelet/pods/abf058d4-f482-47b8-8610-6491dfd318c2/volumes"
Oct 30 13:40:11.488060 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:11.488021    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"]
Oct 30 13:40:12.019087 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.019052    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"]
Oct 30 13:40:12.019944 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.019919    5893 cpu_manager.go:414] "containerMap: removing container" podUID="abf058d4-f482-47b8-8610-6491dfd318c2" containerName="istio-proxy"
Oct 30 13:40:12.019944 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.019944    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf058d4-f482-47b8-8610-6491dfd318c2" containerName="istio-proxy"
Oct 30 13:40:12.020156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.020010    5893 cpu_manager.go:414] "containerMap: removing container" podUID="00423102-32b5-4a2a-8f6d-e93309aae3b6" containerName="maas-api"
Oct 30 13:40:12.020156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.020017    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00423102-32b5-4a2a-8f6d-e93309aae3b6" containerName="maas-api"
Oct 30 13:40:12.020156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.020036    5893 cpu_manager.go:414] "containerMap: removing container" podUID="262cbe09-e912-4db9-b5ea-3d4ddbd073f5" containerName="payload-pre-processing"
Oct 30 13:40:12.020156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.020042    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="262cbe09-e912-4db9-b5ea-3d4ddbd073f5" containerName="payload-pre-processing"
Oct 30 13:40:12.020156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.020068    5893 cpu_manager.go:414] "containerMap: removing container" podUID="25b93077-be04-4210-9cd9-c4b1f182a1d1" containerName="maas-api"
Oct 30 13:40:12.020156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.020075    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b93077-be04-4210-9cd9-c4b1f182a1d1" containerName="maas-api"
Oct 30 13:40:12.020156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.020088    5893 cpu_manager.go:414] "containerMap: removing container" podUID="6311bfd1-218e-4fe6-a79c-74f3d127e805" containerName="maas-api"
Oct 30 13:40:12.020156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.020095    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6311bfd1-218e-4fe6-a79c-74f3d127e805" containerName="maas-api"
Oct 30 13:40:12.020577 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.020231    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="00423102-32b5-4a2a-8f6d-e93309aae3b6" containerName="maas-api"
Oct 30 13:40:12.020577 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.020239    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf058d4-f482-47b8-8610-6491dfd318c2" containerName="istio-proxy"
Oct 30 13:40:12.020577 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.020247    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="262cbe09-e912-4db9-b5ea-3d4ddbd073f5" containerName="payload-pre-processing"
Oct 30 13:40:12.020577 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.020258    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b93077-be04-4210-9cd9-c4b1f182a1d1" containerName="maas-api"
Oct 30 13:40:12.020577 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.020263    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="6311bfd1-218e-4fe6-a79c-74f3d127e805" containerName="maas-api"
Oct 30 13:40:12.020577 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.020290    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf058d4-f482-47b8-8610-6491dfd318c2" containerName="istio-proxy"
Oct 30 13:40:12.025491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.025467    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"
Oct 30 13:40:12.047637 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.047613    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-87e742-dockercfg-7c6gd\""
Oct 30 13:40:12.060888 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.060858    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"]
Oct 30 13:40:12.101611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.101580    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-87e742\""
Oct 30 13:40:12.121284 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.121237    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqbvf\" (UniqueName: \"kubernetes.io/projected/4f2aeac9-60f1-4419-bb4e-65de389c0185-kube-api-access-lqbvf\") pod \"payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8\" (UID: \"4f2aeac9-60f1-4419-bb4e-65de389c0185\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"
Oct 30 13:40:12.121525 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.121505    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2aeac9-60f1-4419-bb4e-65de389c0185-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8\" (UID: \"4f2aeac9-60f1-4419-bb4e-65de389c0185\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"
Oct 30 13:40:12.222810 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.222775    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2aeac9-60f1-4419-bb4e-65de389c0185-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8\" (UID: \"4f2aeac9-60f1-4419-bb4e-65de389c0185\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"
Oct 30 13:40:12.222994 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.222857    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqbvf\" (UniqueName: \"kubernetes.io/projected/4f2aeac9-60f1-4419-bb4e-65de389c0185-kube-api-access-lqbvf\") pod \"payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8\" (UID: \"4f2aeac9-60f1-4419-bb4e-65de389c0185\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"
Oct 30 13:40:12.223628 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.223607    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2aeac9-60f1-4419-bb4e-65de389c0185-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8\" (UID: \"4f2aeac9-60f1-4419-bb4e-65de389c0185\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"
Oct 30 13:40:12.245046 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.245017    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqbvf\" (UniqueName: \"kubernetes.io/projected/4f2aeac9-60f1-4419-bb4e-65de389c0185-kube-api-access-lqbvf\") pod \"payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8\" (UID: \"4f2aeac9-60f1-4419-bb4e-65de389c0185\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"
Oct 30 13:40:12.340896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.340821    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"
Oct 30 13:40:12.658723 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:12.658695    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"]
Oct 30 13:40:12.661959 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:40:12.661928    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f2aeac9_60f1_4419_bb4e_65de389c0185.slice/crio-d4a89941cb801618ec34716fec9e74316f41856049bda4795249abfe807fdedc WatchSource:0}: Error finding container d4a89941cb801618ec34716fec9e74316f41856049bda4795249abfe807fdedc: Status 404 returned error can't find the container with id d4a89941cb801618ec34716fec9e74316f41856049bda4795249abfe807fdedc
Oct 30 13:40:13.614402 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:13.614365    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8" event={"ID":"4f2aeac9-60f1-4419-bb4e-65de389c0185","Type":"ContainerStarted","Data":"503d12ae14e45ade74b1ccdfd3ad56f1b560d8434c3287886944c9b6f62ff31a"}
Oct 30 13:40:13.614636 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:13.614425    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8" event={"ID":"4f2aeac9-60f1-4419-bb4e-65de389c0185","Type":"ContainerStarted","Data":"d4a89941cb801618ec34716fec9e74316f41856049bda4795249abfe807fdedc"}
Oct 30 13:40:13.614636 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:13.614582    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"
Oct 30 13:40:13.663198 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:13.663154    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8" podStartSLOduration=2.663139717 podStartE2EDuration="2.663139717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:13.659363935 +0000 UTC m=+3255.242793630" watchObservedRunningTime="2026-10-30 13:40:13.663139717 +0000 UTC m=+3255.246569409"
Oct 30 13:40:14.108985 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.108940    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8" podUID="00cbb825-5817-49ad-87b0-7f7d8990facb" containerName="payload-processing" containerID="cri-o://43f7a4662d3deac04ac3a873252b358a90a9dff2ffbedd96b40ae823b52a33c3" gracePeriod=25
Oct 30 13:40:14.147990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.147945    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj" podUID="f2e21998-d091-4ca7-87d6-3dcefdfaa4fb" containerName="payload-pre-processing" containerID="cri-o://6a985c1e2697ef9a84d5e8ac9c5acd6998beabab1cded27cae35448ebd62994b" gracePeriod=25
Oct 30 13:40:14.425135 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.425105    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"
Oct 30 13:40:14.456316 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.456257    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"
Oct 30 13:40:14.552657 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.552627    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00cbb825-5817-49ad-87b0-7f7d8990facb-kube-api-access-cjw9j\" (UniqueName: \"kubernetes.io/projected/00cbb825-5817-49ad-87b0-7f7d8990facb-kube-api-access-cjw9j\") pod \"00cbb825-5817-49ad-87b0-7f7d8990facb\" (UID: \"00cbb825-5817-49ad-87b0-7f7d8990facb\") "
Oct 30 13:40:14.552852 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.552784    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb-kube-api-access-9vvhn\" (UniqueName: \"kubernetes.io/projected/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb-kube-api-access-9vvhn\") pod \"f2e21998-d091-4ca7-87d6-3dcefdfaa4fb\" (UID: \"f2e21998-d091-4ca7-87d6-3dcefdfaa4fb\") "
Oct 30 13:40:14.552921 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.552856    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00cbb825-5817-49ad-87b0-7f7d8990facb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00cbb825-5817-49ad-87b0-7f7d8990facb-plugins-config-volume\") pod \"00cbb825-5817-49ad-87b0-7f7d8990facb\" (UID: \"00cbb825-5817-49ad-87b0-7f7d8990facb\") "
Oct 30 13:40:14.552921 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.552903    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb-plugins-config-volume\") pod \"f2e21998-d091-4ca7-87d6-3dcefdfaa4fb\" (UID: \"f2e21998-d091-4ca7-87d6-3dcefdfaa4fb\") "
Oct 30 13:40:14.554171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.554143    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb-plugins-config-volume" pod "f2e21998-d091-4ca7-87d6-3dcefdfaa4fb" (UID: "f2e21998-d091-4ca7-87d6-3dcefdfaa4fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:14.554793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.554767    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00cbb825-5817-49ad-87b0-7f7d8990facb-plugins-config-volume" pod "00cbb825-5817-49ad-87b0-7f7d8990facb" (UID: "00cbb825-5817-49ad-87b0-7f7d8990facb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:14.555745 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.555719    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb-kube-api-access-9vvhn" pod "f2e21998-d091-4ca7-87d6-3dcefdfaa4fb" (UID: "f2e21998-d091-4ca7-87d6-3dcefdfaa4fb"). InnerVolumeSpecName "kube-api-access-9vvhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:14.556109 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.556091    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00cbb825-5817-49ad-87b0-7f7d8990facb-kube-api-access-cjw9j" pod "00cbb825-5817-49ad-87b0-7f7d8990facb" (UID: "00cbb825-5817-49ad-87b0-7f7d8990facb"). InnerVolumeSpecName "kube-api-access-cjw9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:14.621913 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.621877    5893 generic.go:356] "Generic (PLEG): container finished" podID="00cbb825-5817-49ad-87b0-7f7d8990facb" containerID="43f7a4662d3deac04ac3a873252b358a90a9dff2ffbedd96b40ae823b52a33c3" exitCode=0
Oct 30 13:40:14.622101 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.621930    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"
Oct 30 13:40:14.622101 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.621969    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8" event={"ID":"00cbb825-5817-49ad-87b0-7f7d8990facb","Type":"ContainerDied","Data":"43f7a4662d3deac04ac3a873252b358a90a9dff2ffbedd96b40ae823b52a33c3"}
Oct 30 13:40:14.622101 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.622023    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8" event={"ID":"00cbb825-5817-49ad-87b0-7f7d8990facb","Type":"ContainerDied","Data":"075b9c301adef3d6fddbb8b3d409cbf3b23835d7dae83ce699cb68cfa491f4a9"}
Oct 30 13:40:14.622101 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.622050    5893 scope.go:122] "RemoveContainer" containerID="43f7a4662d3deac04ac3a873252b358a90a9dff2ffbedd96b40ae823b52a33c3"
Oct 30 13:40:14.623492 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.623466    5893 generic.go:356] "Generic (PLEG): container finished" podID="f2e21998-d091-4ca7-87d6-3dcefdfaa4fb" containerID="6a985c1e2697ef9a84d5e8ac9c5acd6998beabab1cded27cae35448ebd62994b" exitCode=0
Oct 30 13:40:14.623632 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.623505    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"
Oct 30 13:40:14.623632 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.623561    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj" event={"ID":"f2e21998-d091-4ca7-87d6-3dcefdfaa4fb","Type":"ContainerDied","Data":"6a985c1e2697ef9a84d5e8ac9c5acd6998beabab1cded27cae35448ebd62994b"}
Oct 30 13:40:14.623632 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.623626    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj" event={"ID":"f2e21998-d091-4ca7-87d6-3dcefdfaa4fb","Type":"ContainerDied","Data":"e324a06fea4a7280de252819772b8a535a8fe5bc0384be4c8802b3fcab4474b9"}
Oct 30 13:40:14.648212 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.648184    5893 scope.go:122] "RemoveContainer" containerID="43f7a4662d3deac04ac3a873252b358a90a9dff2ffbedd96b40ae823b52a33c3"
Oct 30 13:40:14.648629 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:14.648602    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f7a4662d3deac04ac3a873252b358a90a9dff2ffbedd96b40ae823b52a33c3\": container with ID starting with 43f7a4662d3deac04ac3a873252b358a90a9dff2ffbedd96b40ae823b52a33c3 not found: ID does not exist" containerID="43f7a4662d3deac04ac3a873252b358a90a9dff2ffbedd96b40ae823b52a33c3"
Oct 30 13:40:14.648747 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.648641    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f7a4662d3deac04ac3a873252b358a90a9dff2ffbedd96b40ae823b52a33c3"} err="failed to get container status \"43f7a4662d3deac04ac3a873252b358a90a9dff2ffbedd96b40ae823b52a33c3\": rpc error: code = NotFound desc = could not find container \"43f7a4662d3deac04ac3a873252b358a90a9dff2ffbedd96b40ae823b52a33c3\": container with ID starting with 43f7a4662d3deac04ac3a873252b358a90a9dff2ffbedd96b40ae823b52a33c3 not found: ID does not exist"
Oct 30 13:40:14.648747 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.648668    5893 scope.go:122] "RemoveContainer" containerID="6a985c1e2697ef9a84d5e8ac9c5acd6998beabab1cded27cae35448ebd62994b"
Oct 30 13:40:14.655907 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.655880    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vvhn\" (UniqueName: \"kubernetes.io/projected/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb-kube-api-access-9vvhn\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:14.656114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.655911    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00cbb825-5817-49ad-87b0-7f7d8990facb-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:14.656114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.655925    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:14.656114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.655939    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjw9j\" (UniqueName: \"kubernetes.io/projected/00cbb825-5817-49ad-87b0-7f7d8990facb-kube-api-access-cjw9j\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:14.659317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.659253    5893 scope.go:122] "RemoveContainer" containerID="6a985c1e2697ef9a84d5e8ac9c5acd6998beabab1cded27cae35448ebd62994b"
Oct 30 13:40:14.659596 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:14.659579    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a985c1e2697ef9a84d5e8ac9c5acd6998beabab1cded27cae35448ebd62994b\": container with ID starting with 6a985c1e2697ef9a84d5e8ac9c5acd6998beabab1cded27cae35448ebd62994b not found: ID does not exist" containerID="6a985c1e2697ef9a84d5e8ac9c5acd6998beabab1cded27cae35448ebd62994b"
Oct 30 13:40:14.659661 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.659604    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a985c1e2697ef9a84d5e8ac9c5acd6998beabab1cded27cae35448ebd62994b"} err="failed to get container status \"6a985c1e2697ef9a84d5e8ac9c5acd6998beabab1cded27cae35448ebd62994b\": rpc error: code = NotFound desc = could not find container \"6a985c1e2697ef9a84d5e8ac9c5acd6998beabab1cded27cae35448ebd62994b\": container with ID starting with 6a985c1e2697ef9a84d5e8ac9c5acd6998beabab1cded27cae35448ebd62994b not found: ID does not exist"
Oct 30 13:40:14.685604 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.685545    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"]
Oct 30 13:40:14.688643 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.688618    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-3e8ee4-86549db59-92djj"]
Oct 30 13:40:14.719586 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.719553    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"]
Oct 30 13:40:14.724947 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:14.724921    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-3e8ee4-57f797b98-p9nl8"]
Oct 30 13:40:16.532124 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:16.532089    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00cbb825-5817-49ad-87b0-7f7d8990facb" path="/var/lib/kubelet/pods/00cbb825-5817-49ad-87b0-7f7d8990facb/volumes"
Oct 30 13:40:16.532558 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:16.532544    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2e21998-d091-4ca7-87d6-3dcefdfaa4fb" path="/var/lib/kubelet/pods/f2e21998-d091-4ca7-87d6-3dcefdfaa4fb/volumes"
Oct 30 13:40:18.037257 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:18.037226    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"]
Oct 30 13:40:18.070753 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:18.070717    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"]
Oct 30 13:40:18.647840 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:18.647793    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:40:19.675708 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:19.675667    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main" containerID="cri-o://7be1a6c7d20dfee28530bb682ec845b45a40fe01d44b76b8cb2e853ee11312b4" gracePeriod=45
Oct 30 13:40:19.958918 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:19.958885    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:40:20.018986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.018949    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04bee432-e371-4029-91c6-d8331e62422e-tls-certs\" (UniqueName: \"kubernetes.io/secret/04bee432-e371-4029-91c6-d8331e62422e-tls-certs\") pod \"04bee432-e371-4029-91c6-d8331e62422e\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") "
Oct 30 13:40:20.019171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.019059    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04bee432-e371-4029-91c6-d8331e62422e-kube-api-access-sprgw\" (UniqueName: \"kubernetes.io/projected/04bee432-e371-4029-91c6-d8331e62422e-kube-api-access-sprgw\") pod \"04bee432-e371-4029-91c6-d8331e62422e\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") "
Oct 30 13:40:20.019171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.019126    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-home\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-home\") pod \"04bee432-e371-4029-91c6-d8331e62422e\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") "
Oct 30 13:40:20.019171 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.019165    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-dshm\") pod \"04bee432-e371-4029-91c6-d8331e62422e\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") "
Oct 30 13:40:20.019410 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.019245    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-tmp-dir\") pod \"04bee432-e371-4029-91c6-d8331e62422e\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") "
Oct 30 13:40:20.019410 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.019289    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-model-cache\") pod \"04bee432-e371-4029-91c6-d8331e62422e\" (UID: \"04bee432-e371-4029-91c6-d8331e62422e\") "
Oct 30 13:40:20.019525 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.019491    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-home" pod "04bee432-e371-4029-91c6-d8331e62422e" (UID: "04bee432-e371-4029-91c6-d8331e62422e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:20.019826 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.019797    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-model-cache" pod "04bee432-e371-4029-91c6-d8331e62422e" (UID: "04bee432-e371-4029-91c6-d8331e62422e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:20.020092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.019941    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-tmp-dir" pod "04bee432-e371-4029-91c6-d8331e62422e" (UID: "04bee432-e371-4029-91c6-d8331e62422e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:20.020092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.020081    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:20.020225 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.020128    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:20.020304 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.020220    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:20.022449 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.022421    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bee432-e371-4029-91c6-d8331e62422e-tls-certs" pod "04bee432-e371-4029-91c6-d8331e62422e" (UID: "04bee432-e371-4029-91c6-d8331e62422e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:20.022555 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.022464    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bee432-e371-4029-91c6-d8331e62422e-kube-api-access-sprgw" pod "04bee432-e371-4029-91c6-d8331e62422e" (UID: "04bee432-e371-4029-91c6-d8331e62422e"). InnerVolumeSpecName "kube-api-access-sprgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:20.023349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.023318    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-dshm" pod "04bee432-e371-4029-91c6-d8331e62422e" (UID: "04bee432-e371-4029-91c6-d8331e62422e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:20.121482 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.121449    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04bee432-e371-4029-91c6-d8331e62422e-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:20.121482 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.121478    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04bee432-e371-4029-91c6-d8331e62422e-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:20.121482 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.121489    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sprgw\" (UniqueName: \"kubernetes.io/projected/04bee432-e371-4029-91c6-d8331e62422e-kube-api-access-sprgw\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:20.665692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.665650    5893 generic.go:356] "Generic (PLEG): container finished" podID="04bee432-e371-4029-91c6-d8331e62422e" containerID="7be1a6c7d20dfee28530bb682ec845b45a40fe01d44b76b8cb2e853ee11312b4" exitCode=0
Oct 30 13:40:20.665892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.665704    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"
Oct 30 13:40:20.665892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.665728    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" event={"ID":"04bee432-e371-4029-91c6-d8331e62422e","Type":"ContainerDied","Data":"7be1a6c7d20dfee28530bb682ec845b45a40fe01d44b76b8cb2e853ee11312b4"}
Oct 30 13:40:20.665892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.665778    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7" event={"ID":"04bee432-e371-4029-91c6-d8331e62422e","Type":"ContainerDied","Data":"b5e2fcb729a685a31c3f5db0c62736ff700749f92520db965f0c94ff62e0e345"}
Oct 30 13:40:20.665892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.665802    5893 scope.go:122] "RemoveContainer" containerID="7be1a6c7d20dfee28530bb682ec845b45a40fe01d44b76b8cb2e853ee11312b4"
Oct 30 13:40:20.679627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.679373    5893 scope.go:122] "RemoveContainer" containerID="7be1a6c7d20dfee28530bb682ec845b45a40fe01d44b76b8cb2e853ee11312b4"
Oct 30 13:40:20.679885 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:20.679655    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be1a6c7d20dfee28530bb682ec845b45a40fe01d44b76b8cb2e853ee11312b4\": container with ID starting with 7be1a6c7d20dfee28530bb682ec845b45a40fe01d44b76b8cb2e853ee11312b4 not found: ID does not exist" containerID="7be1a6c7d20dfee28530bb682ec845b45a40fe01d44b76b8cb2e853ee11312b4"
Oct 30 13:40:20.679885 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.679685    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be1a6c7d20dfee28530bb682ec845b45a40fe01d44b76b8cb2e853ee11312b4"} err="failed to get container status \"7be1a6c7d20dfee28530bb682ec845b45a40fe01d44b76b8cb2e853ee11312b4\": rpc error: code = NotFound desc = could not find container \"7be1a6c7d20dfee28530bb682ec845b45a40fe01d44b76b8cb2e853ee11312b4\": container with ID starting with 7be1a6c7d20dfee28530bb682ec845b45a40fe01d44b76b8cb2e853ee11312b4 not found: ID does not exist"
Oct 30 13:40:20.715073 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.715044    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"]
Oct 30 13:40:20.724624 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:20.724599    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c2ce75/simulator-w2-c2ce75-kserve-6bf6ddb59d-7rzz7"]
Oct 30 13:40:22.532983 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:22.532951    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04bee432-e371-4029-91c6-d8331e62422e" path="/var/lib/kubelet/pods/04bee432-e371-4029-91c6-d8331e62422e/volumes"
Oct 30 13:40:23.038115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:23.038073    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z" podUID="478ff6ac-9262-49aa-b3cc-34423527b462" containerName="payload-processing" containerID="cri-o://16ba6f037c674c54513c7ba81c720bed8cac21cd7abf5f8598ee42cc8c55e46e" gracePeriod=25
Oct 30 13:40:23.071356 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:23.071321    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw" podUID="d20b26c6-2e02-47d3-89e7-c4ca3b34da85" containerName="payload-pre-processing" containerID="cri-o://2b607f697dae92f011563370974a3756437387be945b56b46b8fd3702b4a8c31" gracePeriod=25
Oct 30 13:40:23.779016 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:23.778815    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" containerID="cri-o://fd0922579974bf9828f6e990203883a0cc9a03a3cbb03ace79aaabb1d28d72e8" gracePeriod=45
Oct 30 13:40:23.790700 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:23.790665    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": dial tcp 10.134.0.62:8000: connect: connection refused"
Oct 30 13:40:24.058831 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.058800    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:40:24.173880 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.173841    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-home\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-home\") pod \"05284048-51aa-4885-b6a2-5e88d09f0a7a\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") "
Oct 30 13:40:24.174090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.173928    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-tmp-dir\") pod \"05284048-51aa-4885-b6a2-5e88d09f0a7a\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") "
Oct 30 13:40:24.174090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.173997    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05284048-51aa-4885-b6a2-5e88d09f0a7a-kube-api-access-lfjpk\" (UniqueName: \"kubernetes.io/projected/05284048-51aa-4885-b6a2-5e88d09f0a7a-kube-api-access-lfjpk\") pod \"05284048-51aa-4885-b6a2-5e88d09f0a7a\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") "
Oct 30 13:40:24.174090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.174059    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-dshm\") pod \"05284048-51aa-4885-b6a2-5e88d09f0a7a\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") "
Oct 30 13:40:24.174265 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.174110    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05284048-51aa-4885-b6a2-5e88d09f0a7a-tls-certs\" (UniqueName: \"kubernetes.io/secret/05284048-51aa-4885-b6a2-5e88d09f0a7a-tls-certs\") pod \"05284048-51aa-4885-b6a2-5e88d09f0a7a\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") "
Oct 30 13:40:24.174265 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.174191    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-model-cache\") pod \"05284048-51aa-4885-b6a2-5e88d09f0a7a\" (UID: \"05284048-51aa-4885-b6a2-5e88d09f0a7a\") "
Oct 30 13:40:24.175089 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.175046    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-model-cache" pod "05284048-51aa-4885-b6a2-5e88d09f0a7a" (UID: "05284048-51aa-4885-b6a2-5e88d09f0a7a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:24.175227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.175180    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-home" pod "05284048-51aa-4885-b6a2-5e88d09f0a7a" (UID: "05284048-51aa-4885-b6a2-5e88d09f0a7a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:24.175450 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.175429    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-tmp-dir" pod "05284048-51aa-4885-b6a2-5e88d09f0a7a" (UID: "05284048-51aa-4885-b6a2-5e88d09f0a7a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:24.178302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.178157    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-dshm" pod "05284048-51aa-4885-b6a2-5e88d09f0a7a" (UID: "05284048-51aa-4885-b6a2-5e88d09f0a7a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:24.178302 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.178216    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05284048-51aa-4885-b6a2-5e88d09f0a7a-kube-api-access-lfjpk" pod "05284048-51aa-4885-b6a2-5e88d09f0a7a" (UID: "05284048-51aa-4885-b6a2-5e88d09f0a7a"). InnerVolumeSpecName "kube-api-access-lfjpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:24.178526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.178504    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05284048-51aa-4885-b6a2-5e88d09f0a7a-tls-certs" pod "05284048-51aa-4885-b6a2-5e88d09f0a7a" (UID: "05284048-51aa-4885-b6a2-5e88d09f0a7a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:24.276528 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.276307    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:24.276528 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.276341    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05284048-51aa-4885-b6a2-5e88d09f0a7a-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:24.276528 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.276358    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:24.276528 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.276372    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:24.276528 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.276385    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05284048-51aa-4885-b6a2-5e88d09f0a7a-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:24.276528 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.276398    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfjpk\" (UniqueName: \"kubernetes.io/projected/05284048-51aa-4885-b6a2-5e88d09f0a7a-kube-api-access-lfjpk\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:24.627196 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.627165    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"
Oct 30 13:40:24.698141 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.698106    5893 generic.go:356] "Generic (PLEG): container finished" podID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerID="fd0922579974bf9828f6e990203883a0cc9a03a3cbb03ace79aaabb1d28d72e8" exitCode=0
Oct 30 13:40:24.698349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.698194    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" event={"ID":"05284048-51aa-4885-b6a2-5e88d09f0a7a","Type":"ContainerDied","Data":"fd0922579974bf9828f6e990203883a0cc9a03a3cbb03ace79aaabb1d28d72e8"}
Oct 30 13:40:24.698349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.698230    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx" event={"ID":"05284048-51aa-4885-b6a2-5e88d09f0a7a","Type":"ContainerDied","Data":"34c09e9b3f701d44069c3990f68bda0e86c6bf30351f388a8f62fe22e80e79d3"}
Oct 30 13:40:24.698349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.698252    5893 scope.go:122] "RemoveContainer" containerID="fd0922579974bf9828f6e990203883a0cc9a03a3cbb03ace79aaabb1d28d72e8"
Oct 30 13:40:24.698477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.698435    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"
Oct 30 13:40:24.711422 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.711397    5893 scope.go:122] "RemoveContainer" containerID="fd0922579974bf9828f6e990203883a0cc9a03a3cbb03ace79aaabb1d28d72e8"
Oct 30 13:40:24.711734 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:24.711708    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0922579974bf9828f6e990203883a0cc9a03a3cbb03ace79aaabb1d28d72e8\": container with ID starting with fd0922579974bf9828f6e990203883a0cc9a03a3cbb03ace79aaabb1d28d72e8 not found: ID does not exist" containerID="fd0922579974bf9828f6e990203883a0cc9a03a3cbb03ace79aaabb1d28d72e8"
Oct 30 13:40:24.711823 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.711748    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0922579974bf9828f6e990203883a0cc9a03a3cbb03ace79aaabb1d28d72e8"} err="failed to get container status \"fd0922579974bf9828f6e990203883a0cc9a03a3cbb03ace79aaabb1d28d72e8\": rpc error: code = NotFound desc = could not find container \"fd0922579974bf9828f6e990203883a0cc9a03a3cbb03ace79aaabb1d28d72e8\": container with ID starting with fd0922579974bf9828f6e990203883a0cc9a03a3cbb03ace79aaabb1d28d72e8 not found: ID does not exist"
Oct 30 13:40:24.752395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.752366    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"]
Oct 30 13:40:24.764887 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:24.764853    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-8b46c9/distinct-2-w1-8b46c9-kserve-6b96548775-v5jnx"]
Oct 30 13:40:26.509903 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.509839    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" containerID="cri-o://2ac465e80bf7588928f41033ae55c414ce4e9ad2779ae8eab50404096010de7b" gracePeriod=45
Oct 30 13:40:26.531259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.531234    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" path="/var/lib/kubelet/pods/05284048-51aa-4885-b6a2-5e88d09f0a7a/volumes"
Oct 30 13:40:26.723452 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.719471    5893 generic.go:356] "Generic (PLEG): container finished" podID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerID="2ac465e80bf7588928f41033ae55c414ce4e9ad2779ae8eab50404096010de7b" exitCode=0
Oct 30 13:40:26.723452 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.719648    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" event={"ID":"e3feb48e-aee7-4c06-8b5e-972c9da140d9","Type":"ContainerDied","Data":"2ac465e80bf7588928f41033ae55c414ce4e9ad2779ae8eab50404096010de7b"}
Oct 30 13:40:26.831933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.831908    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:40:26.914953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.914803    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-home\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-home\") pod \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") "
Oct 30 13:40:26.914953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.914869    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3feb48e-aee7-4c06-8b5e-972c9da140d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/e3feb48e-aee7-4c06-8b5e-972c9da140d9-tls-certs\") pod \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") "
Oct 30 13:40:26.914953 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.914949    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-dshm\") pod \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") "
Oct 30 13:40:26.915345 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.915028    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-tmp-dir\") pod \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") "
Oct 30 13:40:26.915345 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.915178    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3feb48e-aee7-4c06-8b5e-972c9da140d9-kube-api-access-mr5wm\" (UniqueName: \"kubernetes.io/projected/e3feb48e-aee7-4c06-8b5e-972c9da140d9-kube-api-access-mr5wm\") pod \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") "
Oct 30 13:40:26.915345 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.915238    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-model-cache\") pod \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\" (UID: \"e3feb48e-aee7-4c06-8b5e-972c9da140d9\") "
Oct 30 13:40:26.916023 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.915990    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-model-cache" pod "e3feb48e-aee7-4c06-8b5e-972c9da140d9" (UID: "e3feb48e-aee7-4c06-8b5e-972c9da140d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:26.916312 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.916263    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-tmp-dir" pod "e3feb48e-aee7-4c06-8b5e-972c9da140d9" (UID: "e3feb48e-aee7-4c06-8b5e-972c9da140d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:26.916532 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.916507    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-home" pod "e3feb48e-aee7-4c06-8b5e-972c9da140d9" (UID: "e3feb48e-aee7-4c06-8b5e-972c9da140d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:26.918067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.918040    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-dshm" pod "e3feb48e-aee7-4c06-8b5e-972c9da140d9" (UID: "e3feb48e-aee7-4c06-8b5e-972c9da140d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:26.918794 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.918765    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3feb48e-aee7-4c06-8b5e-972c9da140d9-kube-api-access-mr5wm" pod "e3feb48e-aee7-4c06-8b5e-972c9da140d9" (UID: "e3feb48e-aee7-4c06-8b5e-972c9da140d9"). InnerVolumeSpecName "kube-api-access-mr5wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:26.922737 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:26.922704    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3feb48e-aee7-4c06-8b5e-972c9da140d9-tls-certs" pod "e3feb48e-aee7-4c06-8b5e-972c9da140d9" (UID: "e3feb48e-aee7-4c06-8b5e-972c9da140d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:27.019724 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.017189    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr5wm\" (UniqueName: \"kubernetes.io/projected/e3feb48e-aee7-4c06-8b5e-972c9da140d9-kube-api-access-mr5wm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:27.019724 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.017232    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:27.019724 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.017249    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:27.019724 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.017261    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e3feb48e-aee7-4c06-8b5e-972c9da140d9-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:27.019724 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.017289    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:27.019724 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.017300    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3feb48e-aee7-4c06-8b5e-972c9da140d9-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:27.506240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.506206    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"]
Oct 30 13:40:27.507614 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.507520    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7" podUID="ce692a70-8f35-4154-a801-c53be256f2f3" containerName="maas-api" containerID="cri-o://5ff4fec9798e939547b3a18a3e6b1c5dc25ee53f0d91bcd1f1acdb210c3f7513" gracePeriod=30
Oct 30 13:40:27.731532 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.731483    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:40:27.738730 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.738694    5893 generic.go:356] "Generic (PLEG): container finished" podID="ce692a70-8f35-4154-a801-c53be256f2f3" containerID="5ff4fec9798e939547b3a18a3e6b1c5dc25ee53f0d91bcd1f1acdb210c3f7513" exitCode=0
Oct 30 13:40:27.738899 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.738746    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7" event={"ID":"ce692a70-8f35-4154-a801-c53be256f2f3","Type":"ContainerDied","Data":"5ff4fec9798e939547b3a18a3e6b1c5dc25ee53f0d91bcd1f1acdb210c3f7513"}
Oct 30 13:40:27.740292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.740255    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq" event={"ID":"e3feb48e-aee7-4c06-8b5e-972c9da140d9","Type":"ContainerDied","Data":"57fea9fdff0fd89b96670067115a78c05e4c1b1894eab242e6035b6c0b989f8f"}
Oct 30 13:40:27.740398 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.740325    5893 scope.go:122] "RemoveContainer" containerID="2ac465e80bf7588928f41033ae55c414ce4e9ad2779ae8eab50404096010de7b"
Oct 30 13:40:27.740451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.740413    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"
Oct 30 13:40:27.795513 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.795485    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:40:27.814899 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.814873    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"]
Oct 30 13:40:27.820089 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.820060    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c90b9e/distinct-w0-c90b9e-kserve-c9f8d65-qtvvq"]
Oct 30 13:40:27.934710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.934674    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce692a70-8f35-4154-a801-c53be256f2f3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce692a70-8f35-4154-a801-c53be256f2f3-maas-api-tls\") pod \"ce692a70-8f35-4154-a801-c53be256f2f3\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") "
Oct 30 13:40:27.934875 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.934854    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce692a70-8f35-4154-a801-c53be256f2f3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce692a70-8f35-4154-a801-c53be256f2f3-openshift-service-ca\") pod \"ce692a70-8f35-4154-a801-c53be256f2f3\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") "
Oct 30 13:40:27.934943 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.934921    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce692a70-8f35-4154-a801-c53be256f2f3-kube-api-access-8mpfc\" (UniqueName: \"kubernetes.io/projected/ce692a70-8f35-4154-a801-c53be256f2f3-kube-api-access-8mpfc\") pod \"ce692a70-8f35-4154-a801-c53be256f2f3\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") "
Oct 30 13:40:27.934990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.934964    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce692a70-8f35-4154-a801-c53be256f2f3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce692a70-8f35-4154-a801-c53be256f2f3-metrics-certs\") pod \"ce692a70-8f35-4154-a801-c53be256f2f3\" (UID: \"ce692a70-8f35-4154-a801-c53be256f2f3\") "
Oct 30 13:40:27.935303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.935256    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce692a70-8f35-4154-a801-c53be256f2f3-openshift-service-ca" pod "ce692a70-8f35-4154-a801-c53be256f2f3" (UID: "ce692a70-8f35-4154-a801-c53be256f2f3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:27.935474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.935451    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce692a70-8f35-4154-a801-c53be256f2f3-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:27.937115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.937086    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce692a70-8f35-4154-a801-c53be256f2f3-maas-api-tls" pod "ce692a70-8f35-4154-a801-c53be256f2f3" (UID: "ce692a70-8f35-4154-a801-c53be256f2f3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:27.937215 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.937133    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce692a70-8f35-4154-a801-c53be256f2f3-kube-api-access-8mpfc" pod "ce692a70-8f35-4154-a801-c53be256f2f3" (UID: "ce692a70-8f35-4154-a801-c53be256f2f3"). InnerVolumeSpecName "kube-api-access-8mpfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:27.937215 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:27.937189    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce692a70-8f35-4154-a801-c53be256f2f3-metrics-certs" pod "ce692a70-8f35-4154-a801-c53be256f2f3" (UID: "ce692a70-8f35-4154-a801-c53be256f2f3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:28.036846 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:28.036759    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mpfc\" (UniqueName: \"kubernetes.io/projected/ce692a70-8f35-4154-a801-c53be256f2f3-kube-api-access-8mpfc\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:28.036846 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:28.036787    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce692a70-8f35-4154-a801-c53be256f2f3-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:28.036846 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:28.036797    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce692a70-8f35-4154-a801-c53be256f2f3-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:28.533076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:28.533043    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" path="/var/lib/kubelet/pods/e3feb48e-aee7-4c06-8b5e-972c9da140d9/volumes"
Oct 30 13:40:28.748454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:28.748423    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"
Oct 30 13:40:28.748454 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:28.748441    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7" event={"ID":"ce692a70-8f35-4154-a801-c53be256f2f3","Type":"ContainerDied","Data":"d71e77c2fb8e3818c2c031ee71cf002a0aa8afe51632c884ea200155d43e7a26"}
Oct 30 13:40:28.749003 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:28.748511    5893 scope.go:122] "RemoveContainer" containerID="5ff4fec9798e939547b3a18a3e6b1c5dc25ee53f0d91bcd1f1acdb210c3f7513"
Oct 30 13:40:28.807355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:28.806746    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"]
Oct 30 13:40:28.825365 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:28.823031    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7"]
Oct 30 13:40:28.883798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:28.883761    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z" podUID="478ff6ac-9262-49aa-b3cc-34423527b462" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:28.884023 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:28.883838    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw" podUID="d20b26c6-2e02-47d3-89e7-c4ca3b34da85" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:30.530643 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:30.530606    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce692a70-8f35-4154-a801-c53be256f2f3" path="/var/lib/kubelet/pods/ce692a70-8f35-4154-a801-c53be256f2f3/volumes"
Oct 30 13:40:30.695897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:30.695848    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-0e874e-cfbfc665b-w2kv7" podUID="ce692a70-8f35-4154-a801-c53be256f2f3" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.88:8443/health\": context deadline exceeded"
Oct 30 13:40:35.839286 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:35.839236    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"]
Oct 30 13:40:37.486598 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.486568    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"]
Oct 30 13:40:37.486983 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.486785    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h" podUID="b215f236-b763-4a75-be1c-0edeb29aaae9" containerName="maas-api" containerID="cri-o://8016c30d1fbef9d21f64a54a76f9f949479bf7531014ca9472e545831b9cb1f8" gracePeriod=30
Oct 30 13:40:37.747829 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.747761    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 13:40:37.767525 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.767495    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-maas-api-tls\") pod \"b215f236-b763-4a75-be1c-0edeb29aaae9\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") "
Oct 30 13:40:37.767687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.767554    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-metrics-certs\") pod \"b215f236-b763-4a75-be1c-0edeb29aaae9\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") "
Oct 30 13:40:37.767687 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.767603    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b215f236-b763-4a75-be1c-0edeb29aaae9-kube-api-access-zcrxw\" (UniqueName: \"kubernetes.io/projected/b215f236-b763-4a75-be1c-0edeb29aaae9-kube-api-access-zcrxw\") pod \"b215f236-b763-4a75-be1c-0edeb29aaae9\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") "
Oct 30 13:40:37.767825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.767798    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b215f236-b763-4a75-be1c-0edeb29aaae9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b215f236-b763-4a75-be1c-0edeb29aaae9-openshift-service-ca\") pod \"b215f236-b763-4a75-be1c-0edeb29aaae9\" (UID: \"b215f236-b763-4a75-be1c-0edeb29aaae9\") "
Oct 30 13:40:37.768224 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.768148    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b215f236-b763-4a75-be1c-0edeb29aaae9-openshift-service-ca" pod "b215f236-b763-4a75-be1c-0edeb29aaae9" (UID: "b215f236-b763-4a75-be1c-0edeb29aaae9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:37.768481 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.768462    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b215f236-b763-4a75-be1c-0edeb29aaae9-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:37.769914 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.769886    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-metrics-certs" pod "b215f236-b763-4a75-be1c-0edeb29aaae9" (UID: "b215f236-b763-4a75-be1c-0edeb29aaae9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:37.770022 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.769978    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-maas-api-tls" pod "b215f236-b763-4a75-be1c-0edeb29aaae9" (UID: "b215f236-b763-4a75-be1c-0edeb29aaae9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:37.770081 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.770058    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b215f236-b763-4a75-be1c-0edeb29aaae9-kube-api-access-zcrxw" pod "b215f236-b763-4a75-be1c-0edeb29aaae9" (UID: "b215f236-b763-4a75-be1c-0edeb29aaae9"). InnerVolumeSpecName "kube-api-access-zcrxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:37.814731 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.814690    5893 generic.go:356] "Generic (PLEG): container finished" podID="b215f236-b763-4a75-be1c-0edeb29aaae9" containerID="8016c30d1fbef9d21f64a54a76f9f949479bf7531014ca9472e545831b9cb1f8" exitCode=0
Oct 30 13:40:37.814915 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.814746    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"
Oct 30 13:40:37.814915 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.814770    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h" event={"ID":"b215f236-b763-4a75-be1c-0edeb29aaae9","Type":"ContainerDied","Data":"8016c30d1fbef9d21f64a54a76f9f949479bf7531014ca9472e545831b9cb1f8"}
Oct 30 13:40:37.814915 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.814824    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h" event={"ID":"b215f236-b763-4a75-be1c-0edeb29aaae9","Type":"ContainerDied","Data":"36ce9564bedb5fe3ddb861928c126efa4633ec6533a818d61130bfb42e7e9a75"}
Oct 30 13:40:37.814915 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.814846    5893 scope.go:122] "RemoveContainer" containerID="8016c30d1fbef9d21f64a54a76f9f949479bf7531014ca9472e545831b9cb1f8"
Oct 30 13:40:37.827490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.827469    5893 scope.go:122] "RemoveContainer" containerID="8016c30d1fbef9d21f64a54a76f9f949479bf7531014ca9472e545831b9cb1f8"
Oct 30 13:40:37.827871 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:37.827848    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8016c30d1fbef9d21f64a54a76f9f949479bf7531014ca9472e545831b9cb1f8\": container with ID starting with 8016c30d1fbef9d21f64a54a76f9f949479bf7531014ca9472e545831b9cb1f8 not found: ID does not exist" containerID="8016c30d1fbef9d21f64a54a76f9f949479bf7531014ca9472e545831b9cb1f8"
Oct 30 13:40:37.827959 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.827888    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8016c30d1fbef9d21f64a54a76f9f949479bf7531014ca9472e545831b9cb1f8"} err="failed to get container status \"8016c30d1fbef9d21f64a54a76f9f949479bf7531014ca9472e545831b9cb1f8\": rpc error: code = NotFound desc = could not find container \"8016c30d1fbef9d21f64a54a76f9f949479bf7531014ca9472e545831b9cb1f8\": container with ID starting with 8016c30d1fbef9d21f64a54a76f9f949479bf7531014ca9472e545831b9cb1f8 not found: ID does not exist"
Oct 30 13:40:37.862708 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.862679    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"]
Oct 30 13:40:37.867846 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.867814    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c2ce75-5bb8bc8b75-jqb9h"]
Oct 30 13:40:37.869998 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.869975    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:37.869998 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.869998    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b215f236-b763-4a75-be1c-0edeb29aaae9-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:37.870174 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:37.870007    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcrxw\" (UniqueName: \"kubernetes.io/projected/b215f236-b763-4a75-be1c-0edeb29aaae9-kube-api-access-zcrxw\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:38.084879 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.082437    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"]
Oct 30 13:40:38.084879 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.083624    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx" podUID="df8bcf75-23e1-4a1c-9384-6781cbdf3d38" containerName="maas-api" containerID="cri-o://e8b24e381a99099b89ae342eddaf923522db307d99629731adb5d95599e06ce0" gracePeriod=30
Oct 30 13:40:38.324720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.324695    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 13:40:38.377520 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.376863    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-openshift-service-ca\") pod \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") "
Oct 30 13:40:38.378076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.378051    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-metrics-certs\") pod \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") "
Oct 30 13:40:38.379571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.378207    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-kube-api-access-jpjkz\" (UniqueName: \"kubernetes.io/projected/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-kube-api-access-jpjkz\") pod \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") "
Oct 30 13:40:38.379571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.378349    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-maas-api-tls\") pod \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\" (UID: \"df8bcf75-23e1-4a1c-9384-6781cbdf3d38\") "
Oct 30 13:40:38.380686 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.380652    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-openshift-service-ca" pod "df8bcf75-23e1-4a1c-9384-6781cbdf3d38" (UID: "df8bcf75-23e1-4a1c-9384-6781cbdf3d38"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:38.381591 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.381530    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-metrics-certs" pod "df8bcf75-23e1-4a1c-9384-6781cbdf3d38" (UID: "df8bcf75-23e1-4a1c-9384-6781cbdf3d38"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:38.382660 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.382622    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-maas-api-tls" pod "df8bcf75-23e1-4a1c-9384-6781cbdf3d38" (UID: "df8bcf75-23e1-4a1c-9384-6781cbdf3d38"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:38.383051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.383025    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-kube-api-access-jpjkz" pod "df8bcf75-23e1-4a1c-9384-6781cbdf3d38" (UID: "df8bcf75-23e1-4a1c-9384-6781cbdf3d38"). InnerVolumeSpecName "kube-api-access-jpjkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:38.479752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.479717    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:38.479752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.479750    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpjkz\" (UniqueName: \"kubernetes.io/projected/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-kube-api-access-jpjkz\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:38.479981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.479763    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:38.479981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.479776    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8bcf75-23e1-4a1c-9384-6781cbdf3d38-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:38.545107 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.545068    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b215f236-b763-4a75-be1c-0edeb29aaae9" path="/var/lib/kubelet/pods/b215f236-b763-4a75-be1c-0edeb29aaae9/volumes"
Oct 30 13:40:38.821430 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.821393    5893 generic.go:356] "Generic (PLEG): container finished" podID="df8bcf75-23e1-4a1c-9384-6781cbdf3d38" containerID="e8b24e381a99099b89ae342eddaf923522db307d99629731adb5d95599e06ce0" exitCode=0
Oct 30 13:40:38.821620 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.821445    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"
Oct 30 13:40:38.821620 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.821478    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx" event={"ID":"df8bcf75-23e1-4a1c-9384-6781cbdf3d38","Type":"ContainerDied","Data":"e8b24e381a99099b89ae342eddaf923522db307d99629731adb5d95599e06ce0"}
Oct 30 13:40:38.821620 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.821535    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx" event={"ID":"df8bcf75-23e1-4a1c-9384-6781cbdf3d38","Type":"ContainerDied","Data":"eb353d26ae57aade4552768025aaba289999f59bf20dc6f95bbc4a47d36e7e1c"}
Oct 30 13:40:38.821620 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.821570    5893 scope.go:122] "RemoveContainer" containerID="e8b24e381a99099b89ae342eddaf923522db307d99629731adb5d95599e06ce0"
Oct 30 13:40:38.832129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.832108    5893 scope.go:122] "RemoveContainer" containerID="e8b24e381a99099b89ae342eddaf923522db307d99629731adb5d95599e06ce0"
Oct 30 13:40:38.832469 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:38.832449    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8b24e381a99099b89ae342eddaf923522db307d99629731adb5d95599e06ce0\": container with ID starting with e8b24e381a99099b89ae342eddaf923522db307d99629731adb5d95599e06ce0 not found: ID does not exist" containerID="e8b24e381a99099b89ae342eddaf923522db307d99629731adb5d95599e06ce0"
Oct 30 13:40:38.832533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.832478    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8b24e381a99099b89ae342eddaf923522db307d99629731adb5d95599e06ce0"} err="failed to get container status \"e8b24e381a99099b89ae342eddaf923522db307d99629731adb5d95599e06ce0\": rpc error: code = NotFound desc = could not find container \"e8b24e381a99099b89ae342eddaf923522db307d99629731adb5d95599e06ce0\": container with ID starting with e8b24e381a99099b89ae342eddaf923522db307d99629731adb5d95599e06ce0 not found: ID does not exist"
Oct 30 13:40:38.870885 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.870844    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"]
Oct 30 13:40:38.882928 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.882890    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c90b9e-7d5cc7c5d8-xs7cx"]
Oct 30 13:40:38.884253 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.884220    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw" podUID="d20b26c6-2e02-47d3-89e7-c4ca3b34da85" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:38.884404 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:38.884257    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z" podUID="478ff6ac-9262-49aa-b3cc-34423527b462" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:40:40.532756 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:40.532722    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8bcf75-23e1-4a1c-9384-6781cbdf3d38" path="/var/lib/kubelet/pods/df8bcf75-23e1-4a1c-9384-6781cbdf3d38/volumes"
Oct 30 13:40:42.097697 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:42.097659    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"]
Oct 30 13:40:43.354736 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.354706    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"
Oct 30 13:40:43.400074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.400046    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"
Oct 30 13:40:43.461505 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.461475    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d20b26c6-2e02-47d3-89e7-c4ca3b34da85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20b26c6-2e02-47d3-89e7-c4ca3b34da85-plugins-config-volume\") pod \"d20b26c6-2e02-47d3-89e7-c4ca3b34da85\" (UID: \"d20b26c6-2e02-47d3-89e7-c4ca3b34da85\") "
Oct 30 13:40:43.461692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.461539    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/478ff6ac-9262-49aa-b3cc-34423527b462-kube-api-access-vchdp\" (UniqueName: \"kubernetes.io/projected/478ff6ac-9262-49aa-b3cc-34423527b462-kube-api-access-vchdp\") pod \"478ff6ac-9262-49aa-b3cc-34423527b462\" (UID: \"478ff6ac-9262-49aa-b3cc-34423527b462\") "
Oct 30 13:40:43.461692 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.461570    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20b26c6-2e02-47d3-89e7-c4ca3b34da85-kube-api-access-jqhj6\" (UniqueName: \"kubernetes.io/projected/d20b26c6-2e02-47d3-89e7-c4ca3b34da85-kube-api-access-jqhj6\") pod \"d20b26c6-2e02-47d3-89e7-c4ca3b34da85\" (UID: \"d20b26c6-2e02-47d3-89e7-c4ca3b34da85\") "
Oct 30 13:40:43.461822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.461724    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/478ff6ac-9262-49aa-b3cc-34423527b462-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478ff6ac-9262-49aa-b3cc-34423527b462-plugins-config-volume\") pod \"478ff6ac-9262-49aa-b3cc-34423527b462\" (UID: \"478ff6ac-9262-49aa-b3cc-34423527b462\") "
Oct 30 13:40:43.461969 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.461924    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20b26c6-2e02-47d3-89e7-c4ca3b34da85-plugins-config-volume" pod "d20b26c6-2e02-47d3-89e7-c4ca3b34da85" (UID: "d20b26c6-2e02-47d3-89e7-c4ca3b34da85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:43.462518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.462322    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/478ff6ac-9262-49aa-b3cc-34423527b462-plugins-config-volume" pod "478ff6ac-9262-49aa-b3cc-34423527b462" (UID: "478ff6ac-9262-49aa-b3cc-34423527b462"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:43.462620 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.462607    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/478ff6ac-9262-49aa-b3cc-34423527b462-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:43.462708 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.462626    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20b26c6-2e02-47d3-89e7-c4ca3b34da85-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:43.464045 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.464021    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/478ff6ac-9262-49aa-b3cc-34423527b462-kube-api-access-vchdp" pod "478ff6ac-9262-49aa-b3cc-34423527b462" (UID: "478ff6ac-9262-49aa-b3cc-34423527b462"). InnerVolumeSpecName "kube-api-access-vchdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:43.464175 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.464087    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20b26c6-2e02-47d3-89e7-c4ca3b34da85-kube-api-access-jqhj6" pod "d20b26c6-2e02-47d3-89e7-c4ca3b34da85" (UID: "d20b26c6-2e02-47d3-89e7-c4ca3b34da85"). InnerVolumeSpecName "kube-api-access-jqhj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:43.563890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.563789    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vchdp\" (UniqueName: \"kubernetes.io/projected/478ff6ac-9262-49aa-b3cc-34423527b462-kube-api-access-vchdp\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:43.563890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.563832    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqhj6\" (UniqueName: \"kubernetes.io/projected/d20b26c6-2e02-47d3-89e7-c4ca3b34da85-kube-api-access-jqhj6\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:43.856361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.856254    5893 generic.go:356] "Generic (PLEG): container finished" podID="478ff6ac-9262-49aa-b3cc-34423527b462" containerID="16ba6f037c674c54513c7ba81c720bed8cac21cd7abf5f8598ee42cc8c55e46e" exitCode=0
Oct 30 13:40:43.856361 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.856342    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"
Oct 30 13:40:43.856616 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.856336    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z" event={"ID":"478ff6ac-9262-49aa-b3cc-34423527b462","Type":"ContainerDied","Data":"16ba6f037c674c54513c7ba81c720bed8cac21cd7abf5f8598ee42cc8c55e46e"}
Oct 30 13:40:43.856616 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.856464    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z" event={"ID":"478ff6ac-9262-49aa-b3cc-34423527b462","Type":"ContainerDied","Data":"582787d173fc3b5fdaeec5a1c13a2d8164c54284eb9e350cc6896105cec659eb"}
Oct 30 13:40:43.856616 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.856499    5893 scope.go:122] "RemoveContainer" containerID="16ba6f037c674c54513c7ba81c720bed8cac21cd7abf5f8598ee42cc8c55e46e"
Oct 30 13:40:43.857972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.857944    5893 generic.go:356] "Generic (PLEG): container finished" podID="d20b26c6-2e02-47d3-89e7-c4ca3b34da85" containerID="2b607f697dae92f011563370974a3756437387be945b56b46b8fd3702b4a8c31" exitCode=0
Oct 30 13:40:43.858117 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.858006    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"
Oct 30 13:40:43.858181 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.858126    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw" event={"ID":"d20b26c6-2e02-47d3-89e7-c4ca3b34da85","Type":"ContainerDied","Data":"2b607f697dae92f011563370974a3756437387be945b56b46b8fd3702b4a8c31"}
Oct 30 13:40:43.858181 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.858158    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw" event={"ID":"d20b26c6-2e02-47d3-89e7-c4ca3b34da85","Type":"ContainerDied","Data":"a4377958cc8591296b721147004511dea541542114f2ae7ba372b5f98666aeb1"}
Oct 30 13:40:43.873141 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.873121    5893 scope.go:122] "RemoveContainer" containerID="16ba6f037c674c54513c7ba81c720bed8cac21cd7abf5f8598ee42cc8c55e46e"
Oct 30 13:40:43.873496 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:43.873466    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16ba6f037c674c54513c7ba81c720bed8cac21cd7abf5f8598ee42cc8c55e46e\": container with ID starting with 16ba6f037c674c54513c7ba81c720bed8cac21cd7abf5f8598ee42cc8c55e46e not found: ID does not exist" containerID="16ba6f037c674c54513c7ba81c720bed8cac21cd7abf5f8598ee42cc8c55e46e"
Oct 30 13:40:43.873645 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.873504    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16ba6f037c674c54513c7ba81c720bed8cac21cd7abf5f8598ee42cc8c55e46e"} err="failed to get container status \"16ba6f037c674c54513c7ba81c720bed8cac21cd7abf5f8598ee42cc8c55e46e\": rpc error: code = NotFound desc = could not find container \"16ba6f037c674c54513c7ba81c720bed8cac21cd7abf5f8598ee42cc8c55e46e\": container with ID starting with 16ba6f037c674c54513c7ba81c720bed8cac21cd7abf5f8598ee42cc8c55e46e not found: ID does not exist"
Oct 30 13:40:43.873645 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.873528    5893 scope.go:122] "RemoveContainer" containerID="2b607f697dae92f011563370974a3756437387be945b56b46b8fd3702b4a8c31"
Oct 30 13:40:43.884719 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.884701    5893 scope.go:122] "RemoveContainer" containerID="2b607f697dae92f011563370974a3756437387be945b56b46b8fd3702b4a8c31"
Oct 30 13:40:43.885021 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:43.884998    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b607f697dae92f011563370974a3756437387be945b56b46b8fd3702b4a8c31\": container with ID starting with 2b607f697dae92f011563370974a3756437387be945b56b46b8fd3702b4a8c31 not found: ID does not exist" containerID="2b607f697dae92f011563370974a3756437387be945b56b46b8fd3702b4a8c31"
Oct 30 13:40:43.885134 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.885026    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b607f697dae92f011563370974a3756437387be945b56b46b8fd3702b4a8c31"} err="failed to get container status \"2b607f697dae92f011563370974a3756437387be945b56b46b8fd3702b4a8c31\": rpc error: code = NotFound desc = could not find container \"2b607f697dae92f011563370974a3756437387be945b56b46b8fd3702b4a8c31\": container with ID starting with 2b607f697dae92f011563370974a3756437387be945b56b46b8fd3702b4a8c31 not found: ID does not exist"
Oct 30 13:40:43.914559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.914533    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"]
Oct 30 13:40:43.919259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.919236    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w5-0e874e-5db5cc5df9-ct7fw"]
Oct 30 13:40:43.948787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.948760    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"]
Oct 30 13:40:43.953105 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:43.953078    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-0e874e-f9f47688b-dc64z"]
Oct 30 13:40:44.531578 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:44.531491    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="478ff6ac-9262-49aa-b3cc-34423527b462" path="/var/lib/kubelet/pods/478ff6ac-9262-49aa-b3cc-34423527b462/volumes"
Oct 30 13:40:44.532014 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:44.531964    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d20b26c6-2e02-47d3-89e7-c4ca3b34da85" path="/var/lib/kubelet/pods/d20b26c6-2e02-47d3-89e7-c4ca3b34da85/volumes"
Oct 30 13:40:48.499752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:48.499721    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"]
Oct 30 13:40:48.500290 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:48.499909    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh" podUID="5979a707-99de-4c43-beac-15c684c66fd0" containerName="istio-proxy" containerID="cri-o://dfd9241f780724acd661b9fb0fb487fd672349c59d04b92f2c561d0901dbe6ad" gracePeriod=30
Oct 30 13:40:48.663892 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:48.663829    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:40:49.825871 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:49.825836    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"]
Oct 30 13:40:50.861999 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:50.861954    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" containerID="cri-o://f12c61d5f5ac0059e12469cd6a084375846c399cca6e89f5756c1cee69a19a2f" gracePeriod=45
Oct 30 13:40:50.933504 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:50.933466    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main" probeResult="failure" output="Get \"https://10.134.0.64:8000/ready\": dial tcp 10.134.0.64:8000: connect: connection refused"
Oct 30 13:40:51.158348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.158323    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:40:51.170978 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.170949    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cdd7c3d-c630-4a09-9882-d731177b67af-tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdd7c3d-c630-4a09-9882-d731177b67af-tls-certs\") pod \"1cdd7c3d-c630-4a09-9882-d731177b67af\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") "
Oct 30 13:40:51.171146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.171002    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-tmp-dir\") pod \"1cdd7c3d-c630-4a09-9882-d731177b67af\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") "
Oct 30 13:40:51.171146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.171125    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-dshm\") pod \"1cdd7c3d-c630-4a09-9882-d731177b67af\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") "
Oct 30 13:40:51.171293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.171153    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-model-cache\") pod \"1cdd7c3d-c630-4a09-9882-d731177b67af\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") "
Oct 30 13:40:51.171293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.171230    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cdd7c3d-c630-4a09-9882-d731177b67af-kube-api-access-qlnfs\" (UniqueName: \"kubernetes.io/projected/1cdd7c3d-c630-4a09-9882-d731177b67af-kube-api-access-qlnfs\") pod \"1cdd7c3d-c630-4a09-9882-d731177b67af\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") "
Oct 30 13:40:51.171418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.171300    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-home\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-home\") pod \"1cdd7c3d-c630-4a09-9882-d731177b67af\" (UID: \"1cdd7c3d-c630-4a09-9882-d731177b67af\") "
Oct 30 13:40:51.171418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.171389    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-tmp-dir" pod "1cdd7c3d-c630-4a09-9882-d731177b67af" (UID: "1cdd7c3d-c630-4a09-9882-d731177b67af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:51.171558 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.171528    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-model-cache" pod "1cdd7c3d-c630-4a09-9882-d731177b67af" (UID: "1cdd7c3d-c630-4a09-9882-d731177b67af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:51.171755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.171712    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-home" pod "1cdd7c3d-c630-4a09-9882-d731177b67af" (UID: "1cdd7c3d-c630-4a09-9882-d731177b67af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:51.171907 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.171873    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:51.171907 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.171892    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:51.171907 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.171904    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:51.173865 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.173834    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-dshm" pod "1cdd7c3d-c630-4a09-9882-d731177b67af" (UID: "1cdd7c3d-c630-4a09-9882-d731177b67af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:51.173963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.173862    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cdd7c3d-c630-4a09-9882-d731177b67af-kube-api-access-qlnfs" pod "1cdd7c3d-c630-4a09-9882-d731177b67af" (UID: "1cdd7c3d-c630-4a09-9882-d731177b67af"). InnerVolumeSpecName "kube-api-access-qlnfs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:51.174067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.174049    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cdd7c3d-c630-4a09-9882-d731177b67af-tls-certs" pod "1cdd7c3d-c630-4a09-9882-d731177b67af" (UID: "1cdd7c3d-c630-4a09-9882-d731177b67af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:51.273760 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.273723    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cdd7c3d-c630-4a09-9882-d731177b67af-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:51.273760 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.273762    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlnfs\" (UniqueName: \"kubernetes.io/projected/1cdd7c3d-c630-4a09-9882-d731177b67af-kube-api-access-qlnfs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:51.273961 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.273778    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cdd7c3d-c630-4a09-9882-d731177b67af-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:51.915207 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.915164    5893 generic.go:356] "Generic (PLEG): container finished" podID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerID="f12c61d5f5ac0059e12469cd6a084375846c399cca6e89f5756c1cee69a19a2f" exitCode=0
Oct 30 13:40:51.915773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.915233    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"
Oct 30 13:40:51.915773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.915252    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" event={"ID":"1cdd7c3d-c630-4a09-9882-d731177b67af","Type":"ContainerDied","Data":"f12c61d5f5ac0059e12469cd6a084375846c399cca6e89f5756c1cee69a19a2f"}
Oct 30 13:40:51.915773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.915333    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn" event={"ID":"1cdd7c3d-c630-4a09-9882-d731177b67af","Type":"ContainerDied","Data":"8d38d01ae62db91836571ed200fb388eae872651e08e966083126afe0d344087"}
Oct 30 13:40:51.915773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.915358    5893 scope.go:122] "RemoveContainer" containerID="f12c61d5f5ac0059e12469cd6a084375846c399cca6e89f5756c1cee69a19a2f"
Oct 30 13:40:51.927945 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.927923    5893 scope.go:122] "RemoveContainer" containerID="f12c61d5f5ac0059e12469cd6a084375846c399cca6e89f5756c1cee69a19a2f"
Oct 30 13:40:51.928298 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:51.928248    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f12c61d5f5ac0059e12469cd6a084375846c399cca6e89f5756c1cee69a19a2f\": container with ID starting with f12c61d5f5ac0059e12469cd6a084375846c399cca6e89f5756c1cee69a19a2f not found: ID does not exist" containerID="f12c61d5f5ac0059e12469cd6a084375846c399cca6e89f5756c1cee69a19a2f"
Oct 30 13:40:51.928367 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.928301    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f12c61d5f5ac0059e12469cd6a084375846c399cca6e89f5756c1cee69a19a2f"} err="failed to get container status \"f12c61d5f5ac0059e12469cd6a084375846c399cca6e89f5756c1cee69a19a2f\": rpc error: code = NotFound desc = could not find container \"f12c61d5f5ac0059e12469cd6a084375846c399cca6e89f5756c1cee69a19a2f\": container with ID starting with f12c61d5f5ac0059e12469cd6a084375846c399cca6e89f5756c1cee69a19a2f not found: ID does not exist"
Oct 30 13:40:51.960241 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.960202    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"]
Oct 30 13:40:51.969951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:51.969925    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-dd90cb/distinct-w6-dd90cb-kserve-67598d94c5-z4wjn"]
Oct 30 13:40:52.534108 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:52.534053    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" path="/var/lib/kubelet/pods/1cdd7c3d-c630-4a09-9882-d731177b67af/volumes"
Oct 30 13:40:53.791027 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.791001    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 13:40:53.808739 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.808709    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5979a707-99de-4c43-beac-15c684c66fd0-kube-api-access-b8kjp\" (UniqueName: \"kubernetes.io/projected/5979a707-99de-4c43-beac-15c684c66fd0-kube-api-access-b8kjp\") pod \"5979a707-99de-4c43-beac-15c684c66fd0\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") "
Oct 30 13:40:53.808907 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.808765    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5979a707-99de-4c43-beac-15c684c66fd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5979a707-99de-4c43-beac-15c684c66fd0-istiod-ca-cert\") pod \"5979a707-99de-4c43-beac-15c684c66fd0\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") "
Oct 30 13:40:53.809026 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.808978    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-istio-data\") pod \"5979a707-99de-4c43-beac-15c684c66fd0\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") "
Oct 30 13:40:53.809200 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.809172    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5979a707-99de-4c43-beac-15c684c66fd0-istiod-ca-cert" pod "5979a707-99de-4c43-beac-15c684c66fd0" (UID: "5979a707-99de-4c43-beac-15c684c66fd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:53.809368 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.809305    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-istio-data" pod "5979a707-99de-4c43-beac-15c684c66fd0" (UID: "5979a707-99de-4c43-beac-15c684c66fd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:53.809438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.809383    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-workload-certs\") pod \"5979a707-99de-4c43-beac-15c684c66fd0\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") "
Oct 30 13:40:53.809438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.809408    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5979a707-99de-4c43-beac-15c684c66fd0-istio-token\" (UniqueName: \"kubernetes.io/projected/5979a707-99de-4c43-beac-15c684c66fd0-istio-token\") pod \"5979a707-99de-4c43-beac-15c684c66fd0\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") "
Oct 30 13:40:53.809438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.809435    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5979a707-99de-4c43-beac-15c684c66fd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5979a707-99de-4c43-beac-15c684c66fd0-istio-podinfo\") pod \"5979a707-99de-4c43-beac-15c684c66fd0\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") "
Oct 30 13:40:53.809603 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.809473    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-workload-socket\") pod \"5979a707-99de-4c43-beac-15c684c66fd0\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") "
Oct 30 13:40:53.809603 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.809499    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-istio-envoy\") pod \"5979a707-99de-4c43-beac-15c684c66fd0\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") "
Oct 30 13:40:53.809603 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.809584    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-credential-socket\") pod \"5979a707-99de-4c43-beac-15c684c66fd0\" (UID: \"5979a707-99de-4c43-beac-15c684c66fd0\") "
Oct 30 13:40:53.809939 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.809915    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-credential-socket" pod "5979a707-99de-4c43-beac-15c684c66fd0" (UID: "5979a707-99de-4c43-beac-15c684c66fd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:53.810037 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.809979    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-workload-certs" pod "5979a707-99de-4c43-beac-15c684c66fd0" (UID: "5979a707-99de-4c43-beac-15c684c66fd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:53.810183 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.810162    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5979a707-99de-4c43-beac-15c684c66fd0-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.810354 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.810335    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.810492 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.810479    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.810639 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.810623    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.810904 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.810251    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-workload-socket" pod "5979a707-99de-4c43-beac-15c684c66fd0" (UID: "5979a707-99de-4c43-beac-15c684c66fd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:53.811914 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.811892    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5979a707-99de-4c43-beac-15c684c66fd0-kube-api-access-b8kjp" pod "5979a707-99de-4c43-beac-15c684c66fd0" (UID: "5979a707-99de-4c43-beac-15c684c66fd0"). InnerVolumeSpecName "kube-api-access-b8kjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:53.812681 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.812656    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5979a707-99de-4c43-beac-15c684c66fd0-istio-podinfo" pod "5979a707-99de-4c43-beac-15c684c66fd0" (UID: "5979a707-99de-4c43-beac-15c684c66fd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:40:53.817658 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.817628    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-istio-envoy" pod "5979a707-99de-4c43-beac-15c684c66fd0" (UID: "5979a707-99de-4c43-beac-15c684c66fd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:53.817764 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.817736    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5979a707-99de-4c43-beac-15c684c66fd0-istio-token" pod "5979a707-99de-4c43-beac-15c684c66fd0" (UID: "5979a707-99de-4c43-beac-15c684c66fd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:53.911785 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.911756    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8kjp\" (UniqueName: \"kubernetes.io/projected/5979a707-99de-4c43-beac-15c684c66fd0-kube-api-access-b8kjp\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.911785 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.911784    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5979a707-99de-4c43-beac-15c684c66fd0-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.911785 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.911793    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5979a707-99de-4c43-beac-15c684c66fd0-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.912044 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.911801    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.912044 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.911810    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5979a707-99de-4c43-beac-15c684c66fd0-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.929655 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.929623    5893 generic.go:356] "Generic (PLEG): container finished" podID="5979a707-99de-4c43-beac-15c684c66fd0" containerID="dfd9241f780724acd661b9fb0fb487fd672349c59d04b92f2c561d0901dbe6ad" exitCode=0
Oct 30 13:40:53.929828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.929669    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"
Oct 30 13:40:53.929828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.929682    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh" event={"ID":"5979a707-99de-4c43-beac-15c684c66fd0","Type":"ContainerDied","Data":"dfd9241f780724acd661b9fb0fb487fd672349c59d04b92f2c561d0901dbe6ad"}
Oct 30 13:40:53.929828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.929728    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh" event={"ID":"5979a707-99de-4c43-beac-15c684c66fd0","Type":"ContainerDied","Data":"856ac146eae30d5c87b36651af3b3cd69127ee476f8a351ed648baa7e78fb822"}
Oct 30 13:40:53.929828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.929750    5893 scope.go:122] "RemoveContainer" containerID="dfd9241f780724acd661b9fb0fb487fd672349c59d04b92f2c561d0901dbe6ad"
Oct 30 13:40:53.942035 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.942014    5893 scope.go:122] "RemoveContainer" containerID="dfd9241f780724acd661b9fb0fb487fd672349c59d04b92f2c561d0901dbe6ad"
Oct 30 13:40:53.942396 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:53.942369    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd9241f780724acd661b9fb0fb487fd672349c59d04b92f2c561d0901dbe6ad\": container with ID starting with dfd9241f780724acd661b9fb0fb487fd672349c59d04b92f2c561d0901dbe6ad not found: ID does not exist" containerID="dfd9241f780724acd661b9fb0fb487fd672349c59d04b92f2c561d0901dbe6ad"
Oct 30 13:40:53.942531 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.942405    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd9241f780724acd661b9fb0fb487fd672349c59d04b92f2c561d0901dbe6ad"} err="failed to get container status \"dfd9241f780724acd661b9fb0fb487fd672349c59d04b92f2c561d0901dbe6ad\": rpc error: code = NotFound desc = could not find container \"dfd9241f780724acd661b9fb0fb487fd672349c59d04b92f2c561d0901dbe6ad\": container with ID starting with dfd9241f780724acd661b9fb0fb487fd672349c59d04b92f2c561d0901dbe6ad not found: ID does not exist"
Oct 30 13:40:53.982420 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.982386    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"]
Oct 30 13:40:53.987461 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:53.987433    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c90b9e-openshift-default-778d7f78bf-4f9kh"]
Oct 30 13:40:54.530374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:54.530334    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5979a707-99de-4c43-beac-15c684c66fd0" path="/var/lib/kubelet/pods/5979a707-99de-4c43-beac-15c684c66fd0/volumes"
Oct 30 13:40:54.827203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:54.827112    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8" podUID="4f2aeac9-60f1-4419-bb4e-65de389c0185" containerName="payload-processing" containerID="cri-o://503d12ae14e45ade74b1ccdfd3ad56f1b560d8434c3287886944c9b6f62ff31a" gracePeriod=25
Oct 30 13:40:55.084707 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.084648    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"
Oct 30 13:40:55.124867 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.124832    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f2aeac9-60f1-4419-bb4e-65de389c0185-kube-api-access-lqbvf\" (UniqueName: \"kubernetes.io/projected/4f2aeac9-60f1-4419-bb4e-65de389c0185-kube-api-access-lqbvf\") pod \"4f2aeac9-60f1-4419-bb4e-65de389c0185\" (UID: \"4f2aeac9-60f1-4419-bb4e-65de389c0185\") "
Oct 30 13:40:55.125078 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.125059    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f2aeac9-60f1-4419-bb4e-65de389c0185-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2aeac9-60f1-4419-bb4e-65de389c0185-plugins-config-volume\") pod \"4f2aeac9-60f1-4419-bb4e-65de389c0185\" (UID: \"4f2aeac9-60f1-4419-bb4e-65de389c0185\") "
Oct 30 13:40:55.125506 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.125485    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f2aeac9-60f1-4419-bb4e-65de389c0185-plugins-config-volume" pod "4f2aeac9-60f1-4419-bb4e-65de389c0185" (UID: "4f2aeac9-60f1-4419-bb4e-65de389c0185"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:55.129686 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.129661    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f2aeac9-60f1-4419-bb4e-65de389c0185-kube-api-access-lqbvf" pod "4f2aeac9-60f1-4419-bb4e-65de389c0185" (UID: "4f2aeac9-60f1-4419-bb4e-65de389c0185"). InnerVolumeSpecName "kube-api-access-lqbvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:55.227082 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.227047    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f2aeac9-60f1-4419-bb4e-65de389c0185-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:55.227082 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.227077    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqbvf\" (UniqueName: \"kubernetes.io/projected/4f2aeac9-60f1-4419-bb4e-65de389c0185-kube-api-access-lqbvf\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:55.945406 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.945358    5893 generic.go:356] "Generic (PLEG): container finished" podID="4f2aeac9-60f1-4419-bb4e-65de389c0185" containerID="503d12ae14e45ade74b1ccdfd3ad56f1b560d8434c3287886944c9b6f62ff31a" exitCode=0
Oct 30 13:40:55.945853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.945481    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"
Oct 30 13:40:55.945853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.945485    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8" event={"ID":"4f2aeac9-60f1-4419-bb4e-65de389c0185","Type":"ContainerDied","Data":"503d12ae14e45ade74b1ccdfd3ad56f1b560d8434c3287886944c9b6f62ff31a"}
Oct 30 13:40:55.945853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.945549    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8" event={"ID":"4f2aeac9-60f1-4419-bb4e-65de389c0185","Type":"ContainerDied","Data":"d4a89941cb801618ec34716fec9e74316f41856049bda4795249abfe807fdedc"}
Oct 30 13:40:55.945853 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.945581    5893 scope.go:122] "RemoveContainer" containerID="503d12ae14e45ade74b1ccdfd3ad56f1b560d8434c3287886944c9b6f62ff31a"
Oct 30 13:40:55.957150 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.957135    5893 scope.go:122] "RemoveContainer" containerID="503d12ae14e45ade74b1ccdfd3ad56f1b560d8434c3287886944c9b6f62ff31a"
Oct 30 13:40:55.957487 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:55.957465    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"503d12ae14e45ade74b1ccdfd3ad56f1b560d8434c3287886944c9b6f62ff31a\": container with ID starting with 503d12ae14e45ade74b1ccdfd3ad56f1b560d8434c3287886944c9b6f62ff31a not found: ID does not exist" containerID="503d12ae14e45ade74b1ccdfd3ad56f1b560d8434c3287886944c9b6f62ff31a"
Oct 30 13:40:55.957609 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.957493    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"503d12ae14e45ade74b1ccdfd3ad56f1b560d8434c3287886944c9b6f62ff31a"} err="failed to get container status \"503d12ae14e45ade74b1ccdfd3ad56f1b560d8434c3287886944c9b6f62ff31a\": rpc error: code = NotFound desc = could not find container \"503d12ae14e45ade74b1ccdfd3ad56f1b560d8434c3287886944c9b6f62ff31a\": container with ID starting with 503d12ae14e45ade74b1ccdfd3ad56f1b560d8434c3287886944c9b6f62ff31a not found: ID does not exist"
Oct 30 13:40:55.992081 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:55.992043    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"]
Oct 30 13:40:56.001664 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:56.001639    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-87e742-77fb4fff4d-mhvr8"]
Oct 30 13:40:56.531761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:56.531728    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f2aeac9-60f1-4419-bb4e-65de389c0185" path="/var/lib/kubelet/pods/4f2aeac9-60f1-4419-bb4e-65de389c0185/volumes"
Oct 30 13:40:57.118918 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.118846    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main" containerID="cri-o://f6228c1c9bd637a43df74782e22ab935b62198c152877425d167be1aef24feb8" gracePeriod=45
Oct 30 13:40:57.411881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.411856    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 13:40:57.451464 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.451428    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-tmp-dir\") pod \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") "
Oct 30 13:40:57.451667 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.451485    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-home\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-home\") pod \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") "
Oct 30 13:40:57.451667 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.451564    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35f04e62-3baf-4ce1-b72e-2f6c5973a758-tls-certs\" (UniqueName: \"kubernetes.io/secret/35f04e62-3baf-4ce1-b72e-2f6c5973a758-tls-certs\") pod \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") "
Oct 30 13:40:57.451667 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.451623    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-model-cache\") pod \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") "
Oct 30 13:40:57.451849 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.451825    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35f04e62-3baf-4ce1-b72e-2f6c5973a758-kube-api-access-7jqz5\" (UniqueName: \"kubernetes.io/projected/35f04e62-3baf-4ce1-b72e-2f6c5973a758-kube-api-access-7jqz5\") pod \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") "
Oct 30 13:40:57.451917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.451856    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-tmp-dir" pod "35f04e62-3baf-4ce1-b72e-2f6c5973a758" (UID: "35f04e62-3baf-4ce1-b72e-2f6c5973a758"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:57.451917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.451868    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-home" pod "35f04e62-3baf-4ce1-b72e-2f6c5973a758" (UID: "35f04e62-3baf-4ce1-b72e-2f6c5973a758"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:57.451917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.451882    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-dshm\") pod \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\" (UID: \"35f04e62-3baf-4ce1-b72e-2f6c5973a758\") "
Oct 30 13:40:57.452081 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.452006    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-model-cache" pod "35f04e62-3baf-4ce1-b72e-2f6c5973a758" (UID: "35f04e62-3baf-4ce1-b72e-2f6c5973a758"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:57.452576 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.452449    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:57.452702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.452590    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:57.452702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.452605    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:57.454288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.454252    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35f04e62-3baf-4ce1-b72e-2f6c5973a758-tls-certs" pod "35f04e62-3baf-4ce1-b72e-2f6c5973a758" (UID: "35f04e62-3baf-4ce1-b72e-2f6c5973a758"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:57.454757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.454732    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-dshm" pod "35f04e62-3baf-4ce1-b72e-2f6c5973a758" (UID: "35f04e62-3baf-4ce1-b72e-2f6c5973a758"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:40:57.455431 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.454756    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f04e62-3baf-4ce1-b72e-2f6c5973a758-kube-api-access-7jqz5" pod "35f04e62-3baf-4ce1-b72e-2f6c5973a758" (UID: "35f04e62-3baf-4ce1-b72e-2f6c5973a758"). InnerVolumeSpecName "kube-api-access-7jqz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:57.553848 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.553815    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35f04e62-3baf-4ce1-b72e-2f6c5973a758-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:57.553848 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.553851    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jqz5\" (UniqueName: \"kubernetes.io/projected/35f04e62-3baf-4ce1-b72e-2f6c5973a758-kube-api-access-7jqz5\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:57.554062 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.553866    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35f04e62-3baf-4ce1-b72e-2f6c5973a758-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:57.961324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.961252    5893 generic.go:356] "Generic (PLEG): container finished" podID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerID="f6228c1c9bd637a43df74782e22ab935b62198c152877425d167be1aef24feb8" exitCode=0
Oct 30 13:40:57.961540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.961344    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"
Oct 30 13:40:57.961540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.961342    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" event={"ID":"35f04e62-3baf-4ce1-b72e-2f6c5973a758","Type":"ContainerDied","Data":"f6228c1c9bd637a43df74782e22ab935b62198c152877425d167be1aef24feb8"}
Oct 30 13:40:57.961540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.961461    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p" event={"ID":"35f04e62-3baf-4ce1-b72e-2f6c5973a758","Type":"ContainerDied","Data":"86c29045abac02118a24b45ebd8229e79f8c1844d0e3861b7d68ab07327a1bef"}
Oct 30 13:40:57.961540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.961488    5893 scope.go:122] "RemoveContainer" containerID="f6228c1c9bd637a43df74782e22ab935b62198c152877425d167be1aef24feb8"
Oct 30 13:40:57.980476 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.980451    5893 scope.go:122] "RemoveContainer" containerID="f6228c1c9bd637a43df74782e22ab935b62198c152877425d167be1aef24feb8"
Oct 30 13:40:57.981166 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:57.981133    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6228c1c9bd637a43df74782e22ab935b62198c152877425d167be1aef24feb8\": container with ID starting with f6228c1c9bd637a43df74782e22ab935b62198c152877425d167be1aef24feb8 not found: ID does not exist" containerID="f6228c1c9bd637a43df74782e22ab935b62198c152877425d167be1aef24feb8"
Oct 30 13:40:57.981530 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:57.981175    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6228c1c9bd637a43df74782e22ab935b62198c152877425d167be1aef24feb8"} err="failed to get container status \"f6228c1c9bd637a43df74782e22ab935b62198c152877425d167be1aef24feb8\": rpc error: code = NotFound desc = could not find container \"f6228c1c9bd637a43df74782e22ab935b62198c152877425d167be1aef24feb8\": container with ID starting with f6228c1c9bd637a43df74782e22ab935b62198c152877425d167be1aef24feb8 not found: ID does not exist"
Oct 30 13:40:58.021738 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:58.021702    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"]
Oct 30 13:40:58.031939 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:58.031912    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-7a3603/simulator-w5-7a3603-kserve-5d6fb6f96-bnk4p"]
Oct 30 13:40:58.532003 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:58.531971    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" path="/var/lib/kubelet/pods/35f04e62-3baf-4ce1-b72e-2f6c5973a758/volumes"
Oct 30 13:40:58.863227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:58.863168    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"]
Oct 30 13:40:58.863490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:58.863423    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4" podUID="1529e3f4-294a-4f88-a8e2-438b5d0778c6" containerName="maas-api" containerID="cri-o://f61cb449505b3a88769c94fdd9fc3cc462877dd065319c1fe71d6bdac2979f0d" gracePeriod=30
Oct 30 13:40:59.165729 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.165693    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 13:40:59.273545 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.273502    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1529e3f4-294a-4f88-a8e2-438b5d0778c6-kube-api-access-s8z9d\" (UniqueName: \"kubernetes.io/projected/1529e3f4-294a-4f88-a8e2-438b5d0778c6-kube-api-access-s8z9d\") pod \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") "
Oct 30 13:40:59.273757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.273597    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1529e3f4-294a-4f88-a8e2-438b5d0778c6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1529e3f4-294a-4f88-a8e2-438b5d0778c6-maas-api-tls\") pod \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") "
Oct 30 13:40:59.273757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.273664    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1529e3f4-294a-4f88-a8e2-438b5d0778c6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1529e3f4-294a-4f88-a8e2-438b5d0778c6-openshift-service-ca\") pod \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") "
Oct 30 13:40:59.273896 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.273757    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1529e3f4-294a-4f88-a8e2-438b5d0778c6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1529e3f4-294a-4f88-a8e2-438b5d0778c6-metrics-certs\") pod \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\" (UID: \"1529e3f4-294a-4f88-a8e2-438b5d0778c6\") "
Oct 30 13:40:59.274067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.274043    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1529e3f4-294a-4f88-a8e2-438b5d0778c6-openshift-service-ca" pod "1529e3f4-294a-4f88-a8e2-438b5d0778c6" (UID: "1529e3f4-294a-4f88-a8e2-438b5d0778c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:59.274223 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.274205    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1529e3f4-294a-4f88-a8e2-438b5d0778c6-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:59.275992 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.275965    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1529e3f4-294a-4f88-a8e2-438b5d0778c6-maas-api-tls" pod "1529e3f4-294a-4f88-a8e2-438b5d0778c6" (UID: "1529e3f4-294a-4f88-a8e2-438b5d0778c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:59.276444 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.276422    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1529e3f4-294a-4f88-a8e2-438b5d0778c6-metrics-certs" pod "1529e3f4-294a-4f88-a8e2-438b5d0778c6" (UID: "1529e3f4-294a-4f88-a8e2-438b5d0778c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:40:59.276444 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.276433    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1529e3f4-294a-4f88-a8e2-438b5d0778c6-kube-api-access-s8z9d" pod "1529e3f4-294a-4f88-a8e2-438b5d0778c6" (UID: "1529e3f4-294a-4f88-a8e2-438b5d0778c6"). InnerVolumeSpecName "kube-api-access-s8z9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:59.375891 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.375857    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1529e3f4-294a-4f88-a8e2-438b5d0778c6-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:59.375891 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.375889    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8z9d\" (UniqueName: \"kubernetes.io/projected/1529e3f4-294a-4f88-a8e2-438b5d0778c6-kube-api-access-s8z9d\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:59.375891 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.375901    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1529e3f4-294a-4f88-a8e2-438b5d0778c6-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:40:59.975952 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.975913    5893 generic.go:356] "Generic (PLEG): container finished" podID="1529e3f4-294a-4f88-a8e2-438b5d0778c6" containerID="f61cb449505b3a88769c94fdd9fc3cc462877dd065319c1fe71d6bdac2979f0d" exitCode=0
Oct 30 13:40:59.976490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.975970    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"
Oct 30 13:40:59.976490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.976008    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4" event={"ID":"1529e3f4-294a-4f88-a8e2-438b5d0778c6","Type":"ContainerDied","Data":"f61cb449505b3a88769c94fdd9fc3cc462877dd065319c1fe71d6bdac2979f0d"}
Oct 30 13:40:59.976490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.976073    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4" event={"ID":"1529e3f4-294a-4f88-a8e2-438b5d0778c6","Type":"ContainerDied","Data":"b5270b712f5d1c310733e5827eade6bcc54279e59703617e1a3d94b9f42c91f8"}
Oct 30 13:40:59.976490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.976103    5893 scope.go:122] "RemoveContainer" containerID="f61cb449505b3a88769c94fdd9fc3cc462877dd065319c1fe71d6bdac2979f0d"
Oct 30 13:40:59.990884 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.990867    5893 scope.go:122] "RemoveContainer" containerID="f61cb449505b3a88769c94fdd9fc3cc462877dd065319c1fe71d6bdac2979f0d"
Oct 30 13:40:59.991213 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:40:59.991195    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f61cb449505b3a88769c94fdd9fc3cc462877dd065319c1fe71d6bdac2979f0d\": container with ID starting with f61cb449505b3a88769c94fdd9fc3cc462877dd065319c1fe71d6bdac2979f0d not found: ID does not exist" containerID="f61cb449505b3a88769c94fdd9fc3cc462877dd065319c1fe71d6bdac2979f0d"
Oct 30 13:40:59.991327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:40:59.991222    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f61cb449505b3a88769c94fdd9fc3cc462877dd065319c1fe71d6bdac2979f0d"} err="failed to get container status \"f61cb449505b3a88769c94fdd9fc3cc462877dd065319c1fe71d6bdac2979f0d\": rpc error: code = NotFound desc = could not find container \"f61cb449505b3a88769c94fdd9fc3cc462877dd065319c1fe71d6bdac2979f0d\": container with ID starting with f61cb449505b3a88769c94fdd9fc3cc462877dd065319c1fe71d6bdac2979f0d not found: ID does not exist"
Oct 30 13:41:00.026006 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:00.025919    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"]
Oct 30 13:41:00.034937 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:00.034902    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-dd90cb-6c54d46c58-4mpr4"]
Oct 30 13:41:00.530078 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:00.530048    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1529e3f4-294a-4f88-a8e2-438b5d0778c6" path="/var/lib/kubelet/pods/1529e3f4-294a-4f88-a8e2-438b5d0778c6/volumes"
Oct 30 13:41:01.362622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.362597    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:41:01.419332 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.419304    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:41:01.424849 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.424828    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:41:01.437107 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.437085    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:41:01.459619 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.459579    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"]
Oct 30 13:41:01.460164 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460145    5893 cpu_manager.go:414] "containerMap: removing container" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main"
Oct 30 13:41:01.460267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460166    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main"
Oct 30 13:41:01.460267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460194    5893 cpu_manager.go:414] "containerMap: removing container" podUID="b215f236-b763-4a75-be1c-0edeb29aaae9" containerName="maas-api"
Oct 30 13:41:01.460267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460202    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b215f236-b763-4a75-be1c-0edeb29aaae9" containerName="maas-api"
Oct 30 13:41:01.460267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460216    5893 cpu_manager.go:414] "containerMap: removing container" podUID="df8bcf75-23e1-4a1c-9384-6781cbdf3d38" containerName="maas-api"
Oct 30 13:41:01.460267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460223    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8bcf75-23e1-4a1c-9384-6781cbdf3d38" containerName="maas-api"
Oct 30 13:41:01.460267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460234    5893 cpu_manager.go:414] "containerMap: removing container" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main"
Oct 30 13:41:01.460267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460241    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main"
Oct 30 13:41:01.460267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460251    5893 cpu_manager.go:414] "containerMap: removing container" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main"
Oct 30 13:41:01.460267 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460258    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460300    5893 cpu_manager.go:414] "containerMap: removing container" podUID="478ff6ac-9262-49aa-b3cc-34423527b462" containerName="payload-processing"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460309    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="478ff6ac-9262-49aa-b3cc-34423527b462" containerName="payload-processing"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460321    5893 cpu_manager.go:414] "containerMap: removing container" podUID="f2e21998-d091-4ca7-87d6-3dcefdfaa4fb" containerName="payload-pre-processing"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460328    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2e21998-d091-4ca7-87d6-3dcefdfaa4fb" containerName="payload-pre-processing"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460344    5893 cpu_manager.go:414] "containerMap: removing container" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460351    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460363    5893 cpu_manager.go:414] "containerMap: removing container" podUID="ce692a70-8f35-4154-a801-c53be256f2f3" containerName="maas-api"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460370    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce692a70-8f35-4154-a801-c53be256f2f3" containerName="maas-api"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460391    5893 cpu_manager.go:414] "containerMap: removing container" podUID="5979a707-99de-4c43-beac-15c684c66fd0" containerName="istio-proxy"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460398    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5979a707-99de-4c43-beac-15c684c66fd0" containerName="istio-proxy"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460408    5893 cpu_manager.go:414] "containerMap: removing container" podUID="00cbb825-5817-49ad-87b0-7f7d8990facb" containerName="payload-processing"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460416    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00cbb825-5817-49ad-87b0-7f7d8990facb" containerName="payload-processing"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460439    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d20b26c6-2e02-47d3-89e7-c4ca3b34da85" containerName="payload-pre-processing"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460446    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d20b26c6-2e02-47d3-89e7-c4ca3b34da85" containerName="payload-pre-processing"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460458    5893 cpu_manager.go:414] "containerMap: removing container" podUID="abf058d4-f482-47b8-8610-6491dfd318c2" containerName="istio-proxy"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460465    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf058d4-f482-47b8-8610-6491dfd318c2" containerName="istio-proxy"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460480    5893 cpu_manager.go:414] "containerMap: removing container" podUID="1529e3f4-294a-4f88-a8e2-438b5d0778c6" containerName="maas-api"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460487    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1529e3f4-294a-4f88-a8e2-438b5d0778c6" containerName="maas-api"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460496    5893 cpu_manager.go:414] "containerMap: removing container" podUID="4f2aeac9-60f1-4419-bb4e-65de389c0185" containerName="payload-processing"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460503    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f2aeac9-60f1-4419-bb4e-65de389c0185" containerName="payload-processing"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460529    5893 cpu_manager.go:414] "containerMap: removing container" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460536    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460654    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="00cbb825-5817-49ad-87b0-7f7d8990facb" containerName="payload-processing"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460668    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="5979a707-99de-4c43-beac-15c684c66fd0" containerName="istio-proxy"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460681    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d20b26c6-2e02-47d3-89e7-c4ca3b34da85" containerName="payload-pre-processing"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460694    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="1529e3f4-294a-4f88-a8e2-438b5d0778c6" containerName="maas-api"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460704    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f2aeac9-60f1-4419-bb4e-65de389c0185" containerName="payload-processing"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460713    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3feb48e-aee7-4c06-8b5e-972c9da140d9" containerName="main"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460724    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="04bee432-e371-4029-91c6-d8331e62422e" containerName="main"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460734    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="b215f236-b763-4a75-be1c-0edeb29aaae9" containerName="maas-api"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460743    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8bcf75-23e1-4a1c-9384-6781cbdf3d38" containerName="maas-api"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460753    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="35f04e62-3baf-4ce1-b72e-2f6c5973a758" containerName="main"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460762    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cdd7c3d-c630-4a09-9882-d731177b67af" containerName="main"
Oct 30 13:41:01.460752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460771    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="478ff6ac-9262-49aa-b3cc-34423527b462" containerName="payload-processing"
Oct 30 13:41:01.462425 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460781    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2e21998-d091-4ca7-87d6-3dcefdfaa4fb" containerName="payload-pre-processing"
Oct 30 13:41:01.462425 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460791    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="05284048-51aa-4885-b6a2-5e88d09f0a7a" containerName="main"
Oct 30 13:41:01.462425 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.460799    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce692a70-8f35-4154-a801-c53be256f2f3" containerName="maas-api"
Oct 30 13:41:01.465827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.465808    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"
Oct 30 13:41:01.495180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.495149    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-87e742\""
Oct 30 13:41:01.495385 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.495247    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-87e742-dockercfg-dmt2h\""
Oct 30 13:41:01.497967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.497939    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"]
Oct 30 13:41:01.502767 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.502736    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71220ae-508e-46fa-8093-fcd9c9bb3231-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2\" (UID: \"d71220ae-508e-46fa-8093-fcd9c9bb3231\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"
Oct 30 13:41:01.502928 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.502904    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4759j\" (UniqueName: \"kubernetes.io/projected/d71220ae-508e-46fa-8093-fcd9c9bb3231-kube-api-access-4759j\") pod \"payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2\" (UID: \"d71220ae-508e-46fa-8093-fcd9c9bb3231\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"
Oct 30 13:41:01.604509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.604476    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71220ae-508e-46fa-8093-fcd9c9bb3231-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2\" (UID: \"d71220ae-508e-46fa-8093-fcd9c9bb3231\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"
Oct 30 13:41:01.604732 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.604558    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4759j\" (UniqueName: \"kubernetes.io/projected/d71220ae-508e-46fa-8093-fcd9c9bb3231-kube-api-access-4759j\") pod \"payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2\" (UID: \"d71220ae-508e-46fa-8093-fcd9c9bb3231\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"
Oct 30 13:41:01.605369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.605349    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71220ae-508e-46fa-8093-fcd9c9bb3231-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2\" (UID: \"d71220ae-508e-46fa-8093-fcd9c9bb3231\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"
Oct 30 13:41:01.627512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.627438    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4759j\" (UniqueName: \"kubernetes.io/projected/d71220ae-508e-46fa-8093-fcd9c9bb3231-kube-api-access-4759j\") pod \"payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2\" (UID: \"d71220ae-508e-46fa-8093-fcd9c9bb3231\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"
Oct 30 13:41:01.788440 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:01.788403    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"
Oct 30 13:41:02.097224 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:02.097189    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"]
Oct 30 13:41:03.000871 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:03.000837    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2" event={"ID":"d71220ae-508e-46fa-8093-fcd9c9bb3231","Type":"ContainerStarted","Data":"5bcae2b88592e67eccb26cf19d243f0953670ceb23fc2e86a5a99f4451fdd969"}
Oct 30 13:41:03.001321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:03.000887    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2" event={"ID":"d71220ae-508e-46fa-8093-fcd9c9bb3231","Type":"ContainerStarted","Data":"e58c612c4a234f52736c01c3942874a7429744361dc3f905262e54c1b452d917"}
Oct 30 13:41:03.001321 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:03.000953    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"
Oct 30 13:41:03.039061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:03.039015    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2" podStartSLOduration=2.039002109 podStartE2EDuration="2.039002109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:03.037150674 +0000 UTC m=+3304.620580380" watchObservedRunningTime="2026-10-30 13:41:03.039002109 +0000 UTC m=+3304.622431802"
Oct 30 13:41:07.381331 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:07.381260    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"]
Oct 30 13:41:07.381813 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:07.381586    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl" podUID="c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" containerName="istio-proxy" containerID="cri-o://f49d65f2261c0788a432ee7f3db6a65e19113d9ec72099c90ebda221617f2988" gracePeriod=30
Oct 30 13:41:12.652876 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.652815    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:41:12.746813 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.746782    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-envoy\") pod \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") "
Oct 30 13:41:12.747008 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.746869    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-workload-certs\") pod \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") "
Oct 30 13:41:12.747008 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.746912    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-workload-socket\") pod \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") "
Oct 30 13:41:12.747008 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.746942    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-kube-api-access-6px2v\" (UniqueName: \"kubernetes.io/projected/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-kube-api-access-6px2v\") pod \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") "
Oct 30 13:41:12.747008 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.746987    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-token\" (UniqueName: \"kubernetes.io/projected/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-token\") pod \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") "
Oct 30 13:41:12.747233 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.747111    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-podinfo\") pod \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") "
Oct 30 13:41:12.747314 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.747207    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-credential-socket\") pod \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") "
Oct 30 13:41:12.747385 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.747337    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istiod-ca-cert\") pod \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") "
Oct 30 13:41:12.747385 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.747368    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-data\") pod \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\" (UID: \"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d\") "
Oct 30 13:41:12.747509 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.747470    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-workload-socket" pod "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" (UID: "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:12.747798 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.747772    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-workload-certs" pod "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" (UID: "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:12.749524 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.748024    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-data" pod "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" (UID: "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:12.749524 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.748151    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:12.749524 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.748172    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:12.749524 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.748743    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-credential-socket" pod "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" (UID: "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:12.749524 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.749052    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istiod-ca-cert" pod "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" (UID: "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:12.751557 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.751531    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-envoy" pod "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" (UID: "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:12.751557 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.751536    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-podinfo" pod "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" (UID: "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:41:12.752794 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.752768    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-token" pod "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" (UID: "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:12.753452 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.753421    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-kube-api-access-6px2v" pod "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" (UID: "c80dd06d-424b-4ab3-ba9d-deb0d6ac166d"). InnerVolumeSpecName "kube-api-access-6px2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:12.849165 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.849133    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:12.849165 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.849163    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:12.849165 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.849173    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:12.849165 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.849180    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:12.849470 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.849189    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6px2v\" (UniqueName: \"kubernetes.io/projected/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-kube-api-access-6px2v\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:12.849470 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.849198    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:12.849470 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:12.849206    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:13.057103 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:13.057067    5893 generic.go:356] "Generic (PLEG): container finished" podID="c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" containerID="f49d65f2261c0788a432ee7f3db6a65e19113d9ec72099c90ebda221617f2988" exitCode=0
Oct 30 13:41:13.057306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:13.057123    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl" event={"ID":"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d","Type":"ContainerDied","Data":"f49d65f2261c0788a432ee7f3db6a65e19113d9ec72099c90ebda221617f2988"}
Oct 30 13:41:13.057306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:13.057134    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"
Oct 30 13:41:13.057306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:13.057175    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl" event={"ID":"c80dd06d-424b-4ab3-ba9d-deb0d6ac166d","Type":"ContainerDied","Data":"9b13655e6fde45fe7ba35f3b5f4bf7b69d21b7fe454fb9de1a6dd46c5ec08483"}
Oct 30 13:41:13.057306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:13.057204    5893 scope.go:122] "RemoveContainer" containerID="f49d65f2261c0788a432ee7f3db6a65e19113d9ec72099c90ebda221617f2988"
Oct 30 13:41:13.068571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:13.068547    5893 scope.go:122] "RemoveContainer" containerID="f49d65f2261c0788a432ee7f3db6a65e19113d9ec72099c90ebda221617f2988"
Oct 30 13:41:13.068910 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:41:13.068889    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f49d65f2261c0788a432ee7f3db6a65e19113d9ec72099c90ebda221617f2988\": container with ID starting with f49d65f2261c0788a432ee7f3db6a65e19113d9ec72099c90ebda221617f2988 not found: ID does not exist" containerID="f49d65f2261c0788a432ee7f3db6a65e19113d9ec72099c90ebda221617f2988"
Oct 30 13:41:13.068967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:13.068922    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f49d65f2261c0788a432ee7f3db6a65e19113d9ec72099c90ebda221617f2988"} err="failed to get container status \"f49d65f2261c0788a432ee7f3db6a65e19113d9ec72099c90ebda221617f2988\": rpc error: code = NotFound desc = could not find container \"f49d65f2261c0788a432ee7f3db6a65e19113d9ec72099c90ebda221617f2988\": container with ID starting with f49d65f2261c0788a432ee7f3db6a65e19113d9ec72099c90ebda221617f2988 not found: ID does not exist"
Oct 30 13:41:13.113059 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:13.113031    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"]
Oct 30 13:41:13.122181 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:13.122154    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-dd90cb-openshift-default-597f54ff7-2hswl"]
Oct 30 13:41:14.008436 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:14.008405    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"
Oct 30 13:41:14.532005 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:14.531971    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" path="/var/lib/kubelet/pods/c80dd06d-424b-4ab3-ba9d-deb0d6ac166d/volumes"
Oct 30 13:41:18.646971 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:41:18.646934    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:41:21.378429 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.378396    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"]
Oct 30 13:41:21.378962 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.378609    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98" podUID="d84ed3c0-d5fc-4961-a037-1c20a8704e50" containerName="maas-api" containerID="cri-o://a68bf5abd0d38c39d51558cc09bd88847e52c9f18b4c1d9b947dbf91833148f7" gracePeriod=30
Oct 30 13:41:21.647662 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.647631    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 13:41:21.748242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.748208    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d84ed3c0-d5fc-4961-a037-1c20a8704e50-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d84ed3c0-d5fc-4961-a037-1c20a8704e50-maas-api-tls\") pod \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") "
Oct 30 13:41:21.748464 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.748434    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d84ed3c0-d5fc-4961-a037-1c20a8704e50-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d84ed3c0-d5fc-4961-a037-1c20a8704e50-metrics-certs\") pod \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") "
Oct 30 13:41:21.748537 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.748496    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d84ed3c0-d5fc-4961-a037-1c20a8704e50-kube-api-access-c95f5\" (UniqueName: \"kubernetes.io/projected/d84ed3c0-d5fc-4961-a037-1c20a8704e50-kube-api-access-c95f5\") pod \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") "
Oct 30 13:41:21.748937 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.748597    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d84ed3c0-d5fc-4961-a037-1c20a8704e50-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84ed3c0-d5fc-4961-a037-1c20a8704e50-openshift-service-ca\") pod \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\" (UID: \"d84ed3c0-d5fc-4961-a037-1c20a8704e50\") "
Oct 30 13:41:21.749500 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.749456    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d84ed3c0-d5fc-4961-a037-1c20a8704e50-openshift-service-ca" pod "d84ed3c0-d5fc-4961-a037-1c20a8704e50" (UID: "d84ed3c0-d5fc-4961-a037-1c20a8704e50"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:21.760407 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.756729    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d84ed3c0-d5fc-4961-a037-1c20a8704e50-metrics-certs" pod "d84ed3c0-d5fc-4961-a037-1c20a8704e50" (UID: "d84ed3c0-d5fc-4961-a037-1c20a8704e50"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:21.760407 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.756769    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d84ed3c0-d5fc-4961-a037-1c20a8704e50-kube-api-access-c95f5" pod "d84ed3c0-d5fc-4961-a037-1c20a8704e50" (UID: "d84ed3c0-d5fc-4961-a037-1c20a8704e50"). InnerVolumeSpecName "kube-api-access-c95f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:21.760407 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.756803    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d84ed3c0-d5fc-4961-a037-1c20a8704e50-maas-api-tls" pod "d84ed3c0-d5fc-4961-a037-1c20a8704e50" (UID: "d84ed3c0-d5fc-4961-a037-1c20a8704e50"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:21.851102 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.851013    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d84ed3c0-d5fc-4961-a037-1c20a8704e50-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:21.851102 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.851057    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c95f5\" (UniqueName: \"kubernetes.io/projected/d84ed3c0-d5fc-4961-a037-1c20a8704e50-kube-api-access-c95f5\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:21.851102 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.851074    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d84ed3c0-d5fc-4961-a037-1c20a8704e50-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:21.851102 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:21.851087    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d84ed3c0-d5fc-4961-a037-1c20a8704e50-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:22.116249 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:22.116213    5893 generic.go:356] "Generic (PLEG): container finished" podID="d84ed3c0-d5fc-4961-a037-1c20a8704e50" containerID="a68bf5abd0d38c39d51558cc09bd88847e52c9f18b4c1d9b947dbf91833148f7" exitCode=0
Oct 30 13:41:22.116477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:22.116283    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"
Oct 30 13:41:22.116477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:22.116300    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98" event={"ID":"d84ed3c0-d5fc-4961-a037-1c20a8704e50","Type":"ContainerDied","Data":"a68bf5abd0d38c39d51558cc09bd88847e52c9f18b4c1d9b947dbf91833148f7"}
Oct 30 13:41:22.116477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:22.116362    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98" event={"ID":"d84ed3c0-d5fc-4961-a037-1c20a8704e50","Type":"ContainerDied","Data":"5ef024fd02f2983440d15ece989469783db45a66ec4b104e42f319aa28f61604"}
Oct 30 13:41:22.116477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:22.116394    5893 scope.go:122] "RemoveContainer" containerID="a68bf5abd0d38c39d51558cc09bd88847e52c9f18b4c1d9b947dbf91833148f7"
Oct 30 13:41:22.127725 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:22.127702    5893 scope.go:122] "RemoveContainer" containerID="a68bf5abd0d38c39d51558cc09bd88847e52c9f18b4c1d9b947dbf91833148f7"
Oct 30 13:41:22.128102 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:41:22.128074    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68bf5abd0d38c39d51558cc09bd88847e52c9f18b4c1d9b947dbf91833148f7\": container with ID starting with a68bf5abd0d38c39d51558cc09bd88847e52c9f18b4c1d9b947dbf91833148f7 not found: ID does not exist" containerID="a68bf5abd0d38c39d51558cc09bd88847e52c9f18b4c1d9b947dbf91833148f7"
Oct 30 13:41:22.128289 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:22.128243    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68bf5abd0d38c39d51558cc09bd88847e52c9f18b4c1d9b947dbf91833148f7"} err="failed to get container status \"a68bf5abd0d38c39d51558cc09bd88847e52c9f18b4c1d9b947dbf91833148f7\": rpc error: code = NotFound desc = could not find container \"a68bf5abd0d38c39d51558cc09bd88847e52c9f18b4c1d9b947dbf91833148f7\": container with ID starting with a68bf5abd0d38c39d51558cc09bd88847e52c9f18b4c1d9b947dbf91833148f7 not found: ID does not exist"
Oct 30 13:41:22.167102 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:22.167071    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"]
Oct 30 13:41:22.172111 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:22.172080    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-7a3603-8698649c5c-xcj98"]
Oct 30 13:41:22.531418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:22.531392    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d84ed3c0-d5fc-4961-a037-1c20a8704e50" path="/var/lib/kubelet/pods/d84ed3c0-d5fc-4961-a037-1c20a8704e50/volumes"
Oct 30 13:41:29.494095 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:29.494059    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"]
Oct 30 13:41:29.494633 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:29.494343    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6" podUID="aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" containerName="istio-proxy" containerID="cri-o://93f6d5fa3de04e7c7603df4d31071462b1f91d26918c17e8c6d3b7bcc2ffa646" gracePeriod=30
Oct 30 13:41:30.203441 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.203411    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"]
Oct 30 13:41:30.203672 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.203646    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj" podUID="47ecad36-94c6-4bf6-8738-a5c19fe9ba78" containerName="maas-api" containerID="cri-o://1db3a3eb2f9859df346b70cc3c1de3a239d6da220bd9b211b479fd39a173b023" gracePeriod=30
Oct 30 13:41:30.381261 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.381231    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"]
Oct 30 13:41:30.381564 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.381513    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786" podUID="fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9" containerName="payload-processing" containerID="cri-o://34c72003bd00d222f337d3e1af39ab62b31b5d2e662a2f739dbd3aba43f17ec2" gracePeriod=30
Oct 30 13:41:30.433585 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.433547    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"]
Oct 30 13:41:30.433887 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.433796    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2" podUID="f02e2238-4a09-4cf3-b20a-d82b0aba4fe3" containerName="payload-pre-processing" containerID="cri-o://ab20139048eb3705ef95be13b022f4fe71e741b9885e8b061f84532408549e2f" gracePeriod=30
Oct 30 13:41:30.561506 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.561479    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:41:30.646664 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.645922    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-kube-api-access-zt52p\" (UniqueName: \"kubernetes.io/projected/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-kube-api-access-zt52p\") pod \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") "
Oct 30 13:41:30.646664 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.645970    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-maas-api-tls\") pod \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") "
Oct 30 13:41:30.646664 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.646074    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-metrics-certs\") pod \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") "
Oct 30 13:41:30.646664 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.646096    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-openshift-service-ca\") pod \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\" (UID: \"47ecad36-94c6-4bf6-8738-a5c19fe9ba78\") "
Oct 30 13:41:30.647103 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.647078    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-openshift-service-ca" pod "47ecad36-94c6-4bf6-8738-a5c19fe9ba78" (UID: "47ecad36-94c6-4bf6-8738-a5c19fe9ba78"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:30.649921 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.649892    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-kube-api-access-zt52p" pod "47ecad36-94c6-4bf6-8738-a5c19fe9ba78" (UID: "47ecad36-94c6-4bf6-8738-a5c19fe9ba78"). InnerVolumeSpecName "kube-api-access-zt52p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:30.650415 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.650390    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-maas-api-tls" pod "47ecad36-94c6-4bf6-8738-a5c19fe9ba78" (UID: "47ecad36-94c6-4bf6-8738-a5c19fe9ba78"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:30.650628 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.650606    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-metrics-certs" pod "47ecad36-94c6-4bf6-8738-a5c19fe9ba78" (UID: "47ecad36-94c6-4bf6-8738-a5c19fe9ba78"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:30.712114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.712086    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"
Oct 30 13:41:30.741319 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.741295    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"
Oct 30 13:41:30.747450 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.747424    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9-plugins-config-volume\") pod \"fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9\" (UID: \"fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9\") "
Oct 30 13:41:30.747591 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.747495    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9-kube-api-access-z95nm\" (UniqueName: \"kubernetes.io/projected/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9-kube-api-access-z95nm\") pod \"fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9\" (UID: \"fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9\") "
Oct 30 13:41:30.747829 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.747805    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9-plugins-config-volume" pod "fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9" (UID: "fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:30.748082 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.748065    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt52p\" (UniqueName: \"kubernetes.io/projected/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-kube-api-access-zt52p\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:30.748170 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.748102    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:30.748170 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.748118    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:30.748170 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.748132    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:30.748170 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.748145    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ecad36-94c6-4bf6-8738-a5c19fe9ba78-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:30.751646 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.751598    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9-kube-api-access-z95nm" pod "fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9" (UID: "fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9"). InnerVolumeSpecName "kube-api-access-z95nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:30.848866 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.848809    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3-kube-api-access-hwvmk\" (UniqueName: \"kubernetes.io/projected/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3-kube-api-access-hwvmk\") pod \"f02e2238-4a09-4cf3-b20a-d82b0aba4fe3\" (UID: \"f02e2238-4a09-4cf3-b20a-d82b0aba4fe3\") "
Oct 30 13:41:30.849067 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.849007    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3-plugins-config-volume\") pod \"f02e2238-4a09-4cf3-b20a-d82b0aba4fe3\" (UID: \"f02e2238-4a09-4cf3-b20a-d82b0aba4fe3\") "
Oct 30 13:41:30.849533 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.849507    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z95nm\" (UniqueName: \"kubernetes.io/projected/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9-kube-api-access-z95nm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:30.849646 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.849507    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3-plugins-config-volume" pod "f02e2238-4a09-4cf3-b20a-d82b0aba4fe3" (UID: "f02e2238-4a09-4cf3-b20a-d82b0aba4fe3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:30.851243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.851220    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3-kube-api-access-hwvmk" pod "f02e2238-4a09-4cf3-b20a-d82b0aba4fe3" (UID: "f02e2238-4a09-4cf3-b20a-d82b0aba4fe3"). InnerVolumeSpecName "kube-api-access-hwvmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:30.951108 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.951066    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwvmk\" (UniqueName: \"kubernetes.io/projected/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3-kube-api-access-hwvmk\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:30.951108 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:30.951099    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:31.173931 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.173840    5893 generic.go:356] "Generic (PLEG): container finished" podID="47ecad36-94c6-4bf6-8738-a5c19fe9ba78" containerID="1db3a3eb2f9859df346b70cc3c1de3a239d6da220bd9b211b479fd39a173b023" exitCode=0
Oct 30 13:41:31.173931 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.173900    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"
Oct 30 13:41:31.173931 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.173921    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj" event={"ID":"47ecad36-94c6-4bf6-8738-a5c19fe9ba78","Type":"ContainerDied","Data":"1db3a3eb2f9859df346b70cc3c1de3a239d6da220bd9b211b479fd39a173b023"}
Oct 30 13:41:31.174234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.173996    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj" event={"ID":"47ecad36-94c6-4bf6-8738-a5c19fe9ba78","Type":"ContainerDied","Data":"32d8b6c45d7b39ee450624b66206cb9fdaaeaeb3fb1b5927ddd14c0e1e6321ab"}
Oct 30 13:41:31.174234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.174021    5893 scope.go:122] "RemoveContainer" containerID="1db3a3eb2f9859df346b70cc3c1de3a239d6da220bd9b211b479fd39a173b023"
Oct 30 13:41:31.175548 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.175526    5893 generic.go:356] "Generic (PLEG): container finished" podID="fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9" containerID="34c72003bd00d222f337d3e1af39ab62b31b5d2e662a2f739dbd3aba43f17ec2" exitCode=0
Oct 30 13:41:31.175665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.175583    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"
Oct 30 13:41:31.175665 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.175636    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786" event={"ID":"fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9","Type":"ContainerDied","Data":"34c72003bd00d222f337d3e1af39ab62b31b5d2e662a2f739dbd3aba43f17ec2"}
Oct 30 13:41:31.175758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.175675    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786" event={"ID":"fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9","Type":"ContainerDied","Data":"ab7316e46ff82edf6632645ced5f61781b26076cf4b0038e506183c406401e3b"}
Oct 30 13:41:31.177472 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.177425    5893 generic.go:356] "Generic (PLEG): container finished" podID="f02e2238-4a09-4cf3-b20a-d82b0aba4fe3" containerID="ab20139048eb3705ef95be13b022f4fe71e741b9885e8b061f84532408549e2f" exitCode=0
Oct 30 13:41:31.177648 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.177501    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2" event={"ID":"f02e2238-4a09-4cf3-b20a-d82b0aba4fe3","Type":"ContainerDied","Data":"ab20139048eb3705ef95be13b022f4fe71e741b9885e8b061f84532408549e2f"}
Oct 30 13:41:31.177648 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.177538    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2" event={"ID":"f02e2238-4a09-4cf3-b20a-d82b0aba4fe3","Type":"ContainerDied","Data":"b0a94e089ee51230deb62f57d1a77b05b022e6f6c73ec94e2bd03b9f28337b84"}
Oct 30 13:41:31.177648 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.177615    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"
Oct 30 13:41:31.186528 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.186511    5893 scope.go:122] "RemoveContainer" containerID="1db3a3eb2f9859df346b70cc3c1de3a239d6da220bd9b211b479fd39a173b023"
Oct 30 13:41:31.186846 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:41:31.186827    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db3a3eb2f9859df346b70cc3c1de3a239d6da220bd9b211b479fd39a173b023\": container with ID starting with 1db3a3eb2f9859df346b70cc3c1de3a239d6da220bd9b211b479fd39a173b023 not found: ID does not exist" containerID="1db3a3eb2f9859df346b70cc3c1de3a239d6da220bd9b211b479fd39a173b023"
Oct 30 13:41:31.186911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.186857    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db3a3eb2f9859df346b70cc3c1de3a239d6da220bd9b211b479fd39a173b023"} err="failed to get container status \"1db3a3eb2f9859df346b70cc3c1de3a239d6da220bd9b211b479fd39a173b023\": rpc error: code = NotFound desc = could not find container \"1db3a3eb2f9859df346b70cc3c1de3a239d6da220bd9b211b479fd39a173b023\": container with ID starting with 1db3a3eb2f9859df346b70cc3c1de3a239d6da220bd9b211b479fd39a173b023 not found: ID does not exist"
Oct 30 13:41:31.186911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.186874    5893 scope.go:122] "RemoveContainer" containerID="34c72003bd00d222f337d3e1af39ab62b31b5d2e662a2f739dbd3aba43f17ec2"
Oct 30 13:41:31.195960 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.195945    5893 scope.go:122] "RemoveContainer" containerID="34c72003bd00d222f337d3e1af39ab62b31b5d2e662a2f739dbd3aba43f17ec2"
Oct 30 13:41:31.196216 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:41:31.196199    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34c72003bd00d222f337d3e1af39ab62b31b5d2e662a2f739dbd3aba43f17ec2\": container with ID starting with 34c72003bd00d222f337d3e1af39ab62b31b5d2e662a2f739dbd3aba43f17ec2 not found: ID does not exist" containerID="34c72003bd00d222f337d3e1af39ab62b31b5d2e662a2f739dbd3aba43f17ec2"
Oct 30 13:41:31.196254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.196225    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34c72003bd00d222f337d3e1af39ab62b31b5d2e662a2f739dbd3aba43f17ec2"} err="failed to get container status \"34c72003bd00d222f337d3e1af39ab62b31b5d2e662a2f739dbd3aba43f17ec2\": rpc error: code = NotFound desc = could not find container \"34c72003bd00d222f337d3e1af39ab62b31b5d2e662a2f739dbd3aba43f17ec2\": container with ID starting with 34c72003bd00d222f337d3e1af39ab62b31b5d2e662a2f739dbd3aba43f17ec2 not found: ID does not exist"
Oct 30 13:41:31.196254 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.196241    5893 scope.go:122] "RemoveContainer" containerID="ab20139048eb3705ef95be13b022f4fe71e741b9885e8b061f84532408549e2f"
Oct 30 13:41:31.204991 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.204974    5893 scope.go:122] "RemoveContainer" containerID="ab20139048eb3705ef95be13b022f4fe71e741b9885e8b061f84532408549e2f"
Oct 30 13:41:31.205235 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:41:31.205217    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab20139048eb3705ef95be13b022f4fe71e741b9885e8b061f84532408549e2f\": container with ID starting with ab20139048eb3705ef95be13b022f4fe71e741b9885e8b061f84532408549e2f not found: ID does not exist" containerID="ab20139048eb3705ef95be13b022f4fe71e741b9885e8b061f84532408549e2f"
Oct 30 13:41:31.205415 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.205242    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab20139048eb3705ef95be13b022f4fe71e741b9885e8b061f84532408549e2f"} err="failed to get container status \"ab20139048eb3705ef95be13b022f4fe71e741b9885e8b061f84532408549e2f\": rpc error: code = NotFound desc = could not find container \"ab20139048eb3705ef95be13b022f4fe71e741b9885e8b061f84532408549e2f\": container with ID starting with ab20139048eb3705ef95be13b022f4fe71e741b9885e8b061f84532408549e2f not found: ID does not exist"
Oct 30 13:41:31.233957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.233936    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"]
Oct 30 13:41:31.243633 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.243611    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-ba8145-789844b468-bhhqj"]
Oct 30 13:41:31.267678 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.267649    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"]
Oct 30 13:41:31.271978 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.271956    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ba8145-5d7fc5698d-sp786"]
Oct 30 13:41:31.302887 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.302845    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"]
Oct 30 13:41:31.307101 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:31.307074    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ba8145-f49954cd7-f9jv2"]
Oct 30 13:41:32.531388 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:32.531356    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47ecad36-94c6-4bf6-8738-a5c19fe9ba78" path="/var/lib/kubelet/pods/47ecad36-94c6-4bf6-8738-a5c19fe9ba78/volumes"
Oct 30 13:41:32.531786 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:32.531756    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f02e2238-4a09-4cf3-b20a-d82b0aba4fe3" path="/var/lib/kubelet/pods/f02e2238-4a09-4cf3-b20a-d82b0aba4fe3/volumes"
Oct 30 13:41:32.532100 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:32.532086    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9" path="/var/lib/kubelet/pods/fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9/volumes"
Oct 30 13:41:33.334905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:33.334868    5893 patch_prober.go:29] interesting pod/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:41:33.335090 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:33.334954    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6" podUID="aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:41:34.754056 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.754024    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:41:34.893510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.893419    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istiod-ca-cert\") pod \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") "
Oct 30 13:41:34.893510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.893462    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-kube-api-access-jdlc7\" (UniqueName: \"kubernetes.io/projected/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-kube-api-access-jdlc7\") pod \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") "
Oct 30 13:41:34.893510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.893491    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-credential-socket\") pod \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") "
Oct 30 13:41:34.893757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.893608    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-podinfo\") pod \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") "
Oct 30 13:41:34.893757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.893713    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-data\") pod \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") "
Oct 30 13:41:34.893870 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.893762    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-workload-socket\") pod \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") "
Oct 30 13:41:34.893870 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.893806    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-workload-certs\") pod \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") "
Oct 30 13:41:34.893870 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.893838    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-credential-socket" pod "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" (UID: "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:34.893870 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.893835    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istiod-ca-cert" pod "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" (UID: "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:34.894076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.893871    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-token\" (UniqueName: \"kubernetes.io/projected/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-token\") pod \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") "
Oct 30 13:41:34.894076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.893943    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-envoy\") pod \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\" (UID: \"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe\") "
Oct 30 13:41:34.894076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.894008    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-data" pod "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" (UID: "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:34.894076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.894021    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-workload-socket" pod "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" (UID: "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:34.894317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.894156    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-workload-certs" pod "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" (UID: "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:34.894407 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.894390    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.894456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.894408    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.894456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.894418    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.894456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.894434    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.894456 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.894451    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.896046 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.896020    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-token" pod "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" (UID: "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:34.896172 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.896135    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-envoy" pod "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" (UID: "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:34.896326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.896309    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-podinfo" pod "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" (UID: "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:41:34.896408 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.896356    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-kube-api-access-jdlc7" pod "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" (UID: "aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe"). InnerVolumeSpecName "kube-api-access-jdlc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:34.995580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.995546    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdlc7\" (UniqueName: \"kubernetes.io/projected/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-kube-api-access-jdlc7\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.995580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.995573    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.995580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.995584    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:34.995580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:34.995593    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:35.203927 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:35.203832    5893 generic.go:356] "Generic (PLEG): container finished" podID="aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" containerID="93f6d5fa3de04e7c7603df4d31071462b1f91d26918c17e8c6d3b7bcc2ffa646" exitCode=0
Oct 30 13:41:35.203927 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:35.203901    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"
Oct 30 13:41:35.204173 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:35.203924    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6" event={"ID":"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe","Type":"ContainerDied","Data":"93f6d5fa3de04e7c7603df4d31071462b1f91d26918c17e8c6d3b7bcc2ffa646"}
Oct 30 13:41:35.204173 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:35.203996    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6" event={"ID":"aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe","Type":"ContainerDied","Data":"6fa5e13fbe9e4ef021366c4c49327189cd785c1e5d5d2b315ca3c04b4027e869"}
Oct 30 13:41:35.204173 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:35.204025    5893 scope.go:122] "RemoveContainer" containerID="93f6d5fa3de04e7c7603df4d31071462b1f91d26918c17e8c6d3b7bcc2ffa646"
Oct 30 13:41:35.215924 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:35.215895    5893 scope.go:122] "RemoveContainer" containerID="93f6d5fa3de04e7c7603df4d31071462b1f91d26918c17e8c6d3b7bcc2ffa646"
Oct 30 13:41:35.216241 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:41:35.216216    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93f6d5fa3de04e7c7603df4d31071462b1f91d26918c17e8c6d3b7bcc2ffa646\": container with ID starting with 93f6d5fa3de04e7c7603df4d31071462b1f91d26918c17e8c6d3b7bcc2ffa646 not found: ID does not exist" containerID="93f6d5fa3de04e7c7603df4d31071462b1f91d26918c17e8c6d3b7bcc2ffa646"
Oct 30 13:41:35.216324 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:35.216257    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93f6d5fa3de04e7c7603df4d31071462b1f91d26918c17e8c6d3b7bcc2ffa646"} err="failed to get container status \"93f6d5fa3de04e7c7603df4d31071462b1f91d26918c17e8c6d3b7bcc2ffa646\": rpc error: code = NotFound desc = could not find container \"93f6d5fa3de04e7c7603df4d31071462b1f91d26918c17e8c6d3b7bcc2ffa646\": container with ID starting with 93f6d5fa3de04e7c7603df4d31071462b1f91d26918c17e8c6d3b7bcc2ffa646 not found: ID does not exist"
Oct 30 13:41:35.249114 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:35.249084    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"]
Oct 30 13:41:35.259638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:35.259609    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7a3603-openshift-default-b6494fbc9-tzmm6"]
Oct 30 13:41:36.532650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:36.532621    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" path="/var/lib/kubelet/pods/aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe/volumes"
Oct 30 13:41:38.474939 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:38.474903    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"]
Oct 30 13:41:38.475458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:38.475165    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerName="istio-proxy" containerID="cri-o://53dfc999504f7a52c08ad43badd0abdc57eca93341cd81befea412dfa4b951de" gracePeriod=30
Oct 30 13:41:40.876900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:40.876863    5893 patch_prober.go:29] interesting pod/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:41:40.877349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:40.876920    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:41:43.743486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.743420    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:41:43.898512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.898468    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-envoy\") pod \"6e02b51d-4bfe-4805-b52e-d65419291f1e\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") "
Oct 30 13:41:43.898512 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.898522    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e02b51d-4bfe-4805-b52e-d65419291f1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e02b51d-4bfe-4805-b52e-d65419291f1e-istiod-ca-cert\") pod \"6e02b51d-4bfe-4805-b52e-d65419291f1e\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") "
Oct 30 13:41:43.898757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.898571    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-data\") pod \"6e02b51d-4bfe-4805-b52e-d65419291f1e\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") "
Oct 30 13:41:43.898757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.898612    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-workload-certs\") pod \"6e02b51d-4bfe-4805-b52e-d65419291f1e\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") "
Oct 30 13:41:43.898757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.898645    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-workload-socket\") pod \"6e02b51d-4bfe-4805-b52e-d65419291f1e\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") "
Oct 30 13:41:43.898757 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.898720    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e02b51d-4bfe-4805-b52e-d65419291f1e-kube-api-access-bt5zj\" (UniqueName: \"kubernetes.io/projected/6e02b51d-4bfe-4805-b52e-d65419291f1e-kube-api-access-bt5zj\") pod \"6e02b51d-4bfe-4805-b52e-d65419291f1e\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") "
Oct 30 13:41:43.898971 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.898800    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-credential-socket\") pod \"6e02b51d-4bfe-4805-b52e-d65419291f1e\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") "
Oct 30 13:41:43.898971 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.898832    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-token\" (UniqueName: \"kubernetes.io/projected/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-token\") pod \"6e02b51d-4bfe-4805-b52e-d65419291f1e\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") "
Oct 30 13:41:43.898971 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.898847    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-data" pod "6e02b51d-4bfe-4805-b52e-d65419291f1e" (UID: "6e02b51d-4bfe-4805-b52e-d65419291f1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:43.898971 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.898943    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-podinfo\") pod \"6e02b51d-4bfe-4805-b52e-d65419291f1e\" (UID: \"6e02b51d-4bfe-4805-b52e-d65419291f1e\") "
Oct 30 13:41:43.899203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.899035    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-workload-socket" pod "6e02b51d-4bfe-4805-b52e-d65419291f1e" (UID: "6e02b51d-4bfe-4805-b52e-d65419291f1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:43.899203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.899109    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-credential-socket" pod "6e02b51d-4bfe-4805-b52e-d65419291f1e" (UID: "6e02b51d-4bfe-4805-b52e-d65419291f1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:43.899203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.899141    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e02b51d-4bfe-4805-b52e-d65419291f1e-istiod-ca-cert" pod "6e02b51d-4bfe-4805-b52e-d65419291f1e" (UID: "6e02b51d-4bfe-4805-b52e-d65419291f1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:43.899203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.899176    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-workload-certs" pod "6e02b51d-4bfe-4805-b52e-d65419291f1e" (UID: "6e02b51d-4bfe-4805-b52e-d65419291f1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:43.899565 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.899460    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:43.899662 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.899642    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:43.899728 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.899661    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:43.899728 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.899676    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e02b51d-4bfe-4805-b52e-d65419291f1e-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:43.899728 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.899688    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:43.901153 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.901125    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e02b51d-4bfe-4805-b52e-d65419291f1e-kube-api-access-bt5zj" pod "6e02b51d-4bfe-4805-b52e-d65419291f1e" (UID: "6e02b51d-4bfe-4805-b52e-d65419291f1e"). InnerVolumeSpecName "kube-api-access-bt5zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:43.901320 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.901149    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-token" pod "6e02b51d-4bfe-4805-b52e-d65419291f1e" (UID: "6e02b51d-4bfe-4805-b52e-d65419291f1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:43.901462 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.901448    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-envoy" pod "6e02b51d-4bfe-4805-b52e-d65419291f1e" (UID: "6e02b51d-4bfe-4805-b52e-d65419291f1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:41:43.901592 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:43.901578    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-podinfo" pod "6e02b51d-4bfe-4805-b52e-d65419291f1e" (UID: "6e02b51d-4bfe-4805-b52e-d65419291f1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:41:44.000315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.000217    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt5zj\" (UniqueName: \"kubernetes.io/projected/6e02b51d-4bfe-4805-b52e-d65419291f1e-kube-api-access-bt5zj\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:44.000315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.000246    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:44.000315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.000255    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:44.000315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.000262    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e02b51d-4bfe-4805-b52e-d65419291f1e-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:44.262317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.262207    5893 generic.go:356] "Generic (PLEG): container finished" podID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerID="53dfc999504f7a52c08ad43badd0abdc57eca93341cd81befea412dfa4b951de" exitCode=0
Oct 30 13:41:44.262317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.262258    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"
Oct 30 13:41:44.262317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.262305    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" event={"ID":"6e02b51d-4bfe-4805-b52e-d65419291f1e","Type":"ContainerDied","Data":"53dfc999504f7a52c08ad43badd0abdc57eca93341cd81befea412dfa4b951de"}
Oct 30 13:41:44.262571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.262385    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6" event={"ID":"6e02b51d-4bfe-4805-b52e-d65419291f1e","Type":"ContainerDied","Data":"a51a3ed7cdf67de90cbfe278607fc547ba437163d6bf17c678157b5be3663f0c"}
Oct 30 13:41:44.262571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.262409    5893 scope.go:122] "RemoveContainer" containerID="53dfc999504f7a52c08ad43badd0abdc57eca93341cd81befea412dfa4b951de"
Oct 30 13:41:44.272949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.272930    5893 scope.go:122] "RemoveContainer" containerID="810c3a6e5a613dee41706a5a570f87dc2b5e4c4a47b683e1575b29506ff32813"
Oct 30 13:41:44.282418 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.282398    5893 scope.go:122] "RemoveContainer" containerID="53dfc999504f7a52c08ad43badd0abdc57eca93341cd81befea412dfa4b951de"
Oct 30 13:41:44.282699 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:41:44.282680    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53dfc999504f7a52c08ad43badd0abdc57eca93341cd81befea412dfa4b951de\": container with ID starting with 53dfc999504f7a52c08ad43badd0abdc57eca93341cd81befea412dfa4b951de not found: ID does not exist" containerID="53dfc999504f7a52c08ad43badd0abdc57eca93341cd81befea412dfa4b951de"
Oct 30 13:41:44.282756 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.282709    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53dfc999504f7a52c08ad43badd0abdc57eca93341cd81befea412dfa4b951de"} err="failed to get container status \"53dfc999504f7a52c08ad43badd0abdc57eca93341cd81befea412dfa4b951de\": rpc error: code = NotFound desc = could not find container \"53dfc999504f7a52c08ad43badd0abdc57eca93341cd81befea412dfa4b951de\": container with ID starting with 53dfc999504f7a52c08ad43badd0abdc57eca93341cd81befea412dfa4b951de not found: ID does not exist"
Oct 30 13:41:44.282756 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.282727    5893 scope.go:122] "RemoveContainer" containerID="810c3a6e5a613dee41706a5a570f87dc2b5e4c4a47b683e1575b29506ff32813"
Oct 30 13:41:44.282974 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:41:44.282956    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"810c3a6e5a613dee41706a5a570f87dc2b5e4c4a47b683e1575b29506ff32813\": container with ID starting with 810c3a6e5a613dee41706a5a570f87dc2b5e4c4a47b683e1575b29506ff32813 not found: ID does not exist" containerID="810c3a6e5a613dee41706a5a570f87dc2b5e4c4a47b683e1575b29506ff32813"
Oct 30 13:41:44.283029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.282986    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"810c3a6e5a613dee41706a5a570f87dc2b5e4c4a47b683e1575b29506ff32813"} err="failed to get container status \"810c3a6e5a613dee41706a5a570f87dc2b5e4c4a47b683e1575b29506ff32813\": rpc error: code = NotFound desc = could not find container \"810c3a6e5a613dee41706a5a570f87dc2b5e4c4a47b683e1575b29506ff32813\": container with ID starting with 810c3a6e5a613dee41706a5a570f87dc2b5e4c4a47b683e1575b29506ff32813 not found: ID does not exist"
Oct 30 13:41:44.312450 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.312417    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"]
Oct 30 13:41:44.322164 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.322140    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-ba8145-openshift-default-599d44d7f7-g5lg6"]
Oct 30 13:41:44.529920 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:44.529840    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" path="/var/lib/kubelet/pods/6e02b51d-4bfe-4805-b52e-d65419291f1e/volumes"
Oct 30 13:41:48.645592 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:41:48.645551    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:41:56.031191 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.031152    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"]
Oct 30 13:41:56.031683 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.031447    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2" podUID="d71220ae-508e-46fa-8093-fcd9c9bb3231" containerName="payload-pre-processing" containerID="cri-o://5bcae2b88592e67eccb26cf19d243f0953670ceb23fc2e86a5a99f4451fdd969" gracePeriod=30
Oct 30 13:41:56.307130 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.307067    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"
Oct 30 13:41:56.327095 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.327060    5893 generic.go:356] "Generic (PLEG): container finished" podID="d71220ae-508e-46fa-8093-fcd9c9bb3231" containerID="5bcae2b88592e67eccb26cf19d243f0953670ceb23fc2e86a5a99f4451fdd969" exitCode=0
Oct 30 13:41:56.327306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.327130    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2" event={"ID":"d71220ae-508e-46fa-8093-fcd9c9bb3231","Type":"ContainerDied","Data":"5bcae2b88592e67eccb26cf19d243f0953670ceb23fc2e86a5a99f4451fdd969"}
Oct 30 13:41:56.327306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.327188    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"
Oct 30 13:41:56.327306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.327198    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2" event={"ID":"d71220ae-508e-46fa-8093-fcd9c9bb3231","Type":"ContainerDied","Data":"e58c612c4a234f52736c01c3942874a7429744361dc3f905262e54c1b452d917"}
Oct 30 13:41:56.327306 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.327230    5893 scope.go:122] "RemoveContainer" containerID="5bcae2b88592e67eccb26cf19d243f0953670ceb23fc2e86a5a99f4451fdd969"
Oct 30 13:41:56.328215 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.328188    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71220ae-508e-46fa-8093-fcd9c9bb3231-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71220ae-508e-46fa-8093-fcd9c9bb3231-plugins-config-volume\") pod \"d71220ae-508e-46fa-8093-fcd9c9bb3231\" (UID: \"d71220ae-508e-46fa-8093-fcd9c9bb3231\") "
Oct 30 13:41:56.328391 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.328267    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71220ae-508e-46fa-8093-fcd9c9bb3231-kube-api-access-4759j\" (UniqueName: \"kubernetes.io/projected/d71220ae-508e-46fa-8093-fcd9c9bb3231-kube-api-access-4759j\") pod \"d71220ae-508e-46fa-8093-fcd9c9bb3231\" (UID: \"d71220ae-508e-46fa-8093-fcd9c9bb3231\") "
Oct 30 13:41:56.329670 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.329644    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71220ae-508e-46fa-8093-fcd9c9bb3231-plugins-config-volume" pod "d71220ae-508e-46fa-8093-fcd9c9bb3231" (UID: "d71220ae-508e-46fa-8093-fcd9c9bb3231"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:56.331631 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.331589    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71220ae-508e-46fa-8093-fcd9c9bb3231-kube-api-access-4759j" pod "d71220ae-508e-46fa-8093-fcd9c9bb3231" (UID: "d71220ae-508e-46fa-8093-fcd9c9bb3231"). InnerVolumeSpecName "kube-api-access-4759j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:56.352146 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.352094    5893 scope.go:122] "RemoveContainer" containerID="5bcae2b88592e67eccb26cf19d243f0953670ceb23fc2e86a5a99f4451fdd969"
Oct 30 13:41:56.352706 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:41:56.352625    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bcae2b88592e67eccb26cf19d243f0953670ceb23fc2e86a5a99f4451fdd969\": container with ID starting with 5bcae2b88592e67eccb26cf19d243f0953670ceb23fc2e86a5a99f4451fdd969 not found: ID does not exist" containerID="5bcae2b88592e67eccb26cf19d243f0953670ceb23fc2e86a5a99f4451fdd969"
Oct 30 13:41:56.352706 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.352668    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bcae2b88592e67eccb26cf19d243f0953670ceb23fc2e86a5a99f4451fdd969"} err="failed to get container status \"5bcae2b88592e67eccb26cf19d243f0953670ceb23fc2e86a5a99f4451fdd969\": rpc error: code = NotFound desc = could not find container \"5bcae2b88592e67eccb26cf19d243f0953670ceb23fc2e86a5a99f4451fdd969\": container with ID starting with 5bcae2b88592e67eccb26cf19d243f0953670ceb23fc2e86a5a99f4451fdd969 not found: ID does not exist"
Oct 30 13:41:56.429340 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.429312    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71220ae-508e-46fa-8093-fcd9c9bb3231-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:56.429340 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.429337    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4759j\" (UniqueName: \"kubernetes.io/projected/d71220ae-508e-46fa-8093-fcd9c9bb3231-kube-api-access-4759j\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:41:56.677170 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.677141    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"]
Oct 30 13:41:56.684341 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:56.684315    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-87e742-5d8c8fc75-77lw2"]
Oct 30 13:41:58.529560 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:41:58.529525    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71220ae-508e-46fa-8093-fcd9c9bb3231" path="/var/lib/kubelet/pods/d71220ae-508e-46fa-8093-fcd9c9bb3231/volumes"
Oct 30 13:42:02.016056 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:02.016022    5893 scope.go:122] "RemoveContainer" containerID="457d9b6276efd816731ecf5561cb5fbe208cd49b166648cfb1745fe8575429ff"
Oct 30 13:42:18.646016 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:42:18.645976    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:42:24.906915 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:24.906882    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"]
Oct 30 13:42:24.907430 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:24.907114    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx" podUID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerName="istio-proxy" containerID="cri-o://f67831946436397bd927a8cd94f0f2c3f8e7bfb39fcd601c3e5b6a5966b046b2" gracePeriod=30
Oct 30 13:42:30.164633 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.164607    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:42:30.189773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.189697    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-workload-certs\") pod \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") "
Oct 30 13:42:30.189773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.189759    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-kube-api-access-bcnjd\" (UniqueName: \"kubernetes.io/projected/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-kube-api-access-bcnjd\") pod \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") "
Oct 30 13:42:30.190816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.189787    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-token\" (UniqueName: \"kubernetes.io/projected/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-token\") pod \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") "
Oct 30 13:42:30.190816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.189819    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-workload-socket\") pod \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") "
Oct 30 13:42:30.190816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.189892    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-data\") pod \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") "
Oct 30 13:42:30.190816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.189919    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-credential-socket\") pod \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") "
Oct 30 13:42:30.190816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.189958    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istiod-ca-cert\") pod \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") "
Oct 30 13:42:30.190816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.190013    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-envoy\") pod \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") "
Oct 30 13:42:30.190816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.190064    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-workload-certs" pod "daffe0a0-8e1d-4bbf-a044-67a5f3af103a" (UID: "daffe0a0-8e1d-4bbf-a044-67a5f3af103a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:30.190816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.190322    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-credential-socket" pod "daffe0a0-8e1d-4bbf-a044-67a5f3af103a" (UID: "daffe0a0-8e1d-4bbf-a044-67a5f3af103a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:30.190816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.190410    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istiod-ca-cert" pod "daffe0a0-8e1d-4bbf-a044-67a5f3af103a" (UID: "daffe0a0-8e1d-4bbf-a044-67a5f3af103a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:30.190816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.190431    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.190816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.190449    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.190816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.190455    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-workload-socket" pod "daffe0a0-8e1d-4bbf-a044-67a5f3af103a" (UID: "daffe0a0-8e1d-4bbf-a044-67a5f3af103a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:30.190816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.190514    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-data" pod "daffe0a0-8e1d-4bbf-a044-67a5f3af103a" (UID: "daffe0a0-8e1d-4bbf-a044-67a5f3af103a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:30.192383 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.192347    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-kube-api-access-bcnjd" pod "daffe0a0-8e1d-4bbf-a044-67a5f3af103a" (UID: "daffe0a0-8e1d-4bbf-a044-67a5f3af103a"). InnerVolumeSpecName "kube-api-access-bcnjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:30.192666 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.192642    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-token" pod "daffe0a0-8e1d-4bbf-a044-67a5f3af103a" (UID: "daffe0a0-8e1d-4bbf-a044-67a5f3af103a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:30.192790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.192687    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-envoy" pod "daffe0a0-8e1d-4bbf-a044-67a5f3af103a" (UID: "daffe0a0-8e1d-4bbf-a044-67a5f3af103a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:30.291614 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.291568    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-podinfo\") pod \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\" (UID: \"daffe0a0-8e1d-4bbf-a044-67a5f3af103a\") "
Oct 30 13:42:30.291927 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.291909    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.292015 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.291931    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.292015 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.291944    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.292015 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.291956    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.292015 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.291968    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcnjd\" (UniqueName: \"kubernetes.io/projected/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-kube-api-access-bcnjd\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.292015 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.291981    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.293765 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.293743    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-podinfo" pod "daffe0a0-8e1d-4bbf-a044-67a5f3af103a" (UID: "daffe0a0-8e1d-4bbf-a044-67a5f3af103a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:42:30.392722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.392671    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/daffe0a0-8e1d-4bbf-a044-67a5f3af103a-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:30.501182 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.501150    5893 generic.go:356] "Generic (PLEG): container finished" podID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerID="f67831946436397bd927a8cd94f0f2c3f8e7bfb39fcd601c3e5b6a5966b046b2" exitCode=0
Oct 30 13:42:30.501392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.501204    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"
Oct 30 13:42:30.501392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.501225    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx" event={"ID":"daffe0a0-8e1d-4bbf-a044-67a5f3af103a","Type":"ContainerDied","Data":"f67831946436397bd927a8cd94f0f2c3f8e7bfb39fcd601c3e5b6a5966b046b2"}
Oct 30 13:42:30.501392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.501306    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx" event={"ID":"daffe0a0-8e1d-4bbf-a044-67a5f3af103a","Type":"ContainerDied","Data":"75f6324ec94f88744dc0015ce04921bd499f3ff6750f4b94f28940cf61dc2989"}
Oct 30 13:42:30.501392 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.501339    5893 scope.go:122] "RemoveContainer" containerID="f67831946436397bd927a8cd94f0f2c3f8e7bfb39fcd601c3e5b6a5966b046b2"
Oct 30 13:42:30.512342 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.512325    5893 scope.go:122] "RemoveContainer" containerID="3b51a2e3313b390ba71d9224ab277bbb3367033ee59b1e0646a69722d0debff5"
Oct 30 13:42:30.521370 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.521344    5893 scope.go:122] "RemoveContainer" containerID="f67831946436397bd927a8cd94f0f2c3f8e7bfb39fcd601c3e5b6a5966b046b2"
Oct 30 13:42:30.521640 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:42:30.521619    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67831946436397bd927a8cd94f0f2c3f8e7bfb39fcd601c3e5b6a5966b046b2\": container with ID starting with f67831946436397bd927a8cd94f0f2c3f8e7bfb39fcd601c3e5b6a5966b046b2 not found: ID does not exist" containerID="f67831946436397bd927a8cd94f0f2c3f8e7bfb39fcd601c3e5b6a5966b046b2"
Oct 30 13:42:30.521722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.521655    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67831946436397bd927a8cd94f0f2c3f8e7bfb39fcd601c3e5b6a5966b046b2"} err="failed to get container status \"f67831946436397bd927a8cd94f0f2c3f8e7bfb39fcd601c3e5b6a5966b046b2\": rpc error: code = NotFound desc = could not find container \"f67831946436397bd927a8cd94f0f2c3f8e7bfb39fcd601c3e5b6a5966b046b2\": container with ID starting with f67831946436397bd927a8cd94f0f2c3f8e7bfb39fcd601c3e5b6a5966b046b2 not found: ID does not exist"
Oct 30 13:42:30.521722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.521679    5893 scope.go:122] "RemoveContainer" containerID="3b51a2e3313b390ba71d9224ab277bbb3367033ee59b1e0646a69722d0debff5"
Oct 30 13:42:30.521939 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:42:30.521916    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b51a2e3313b390ba71d9224ab277bbb3367033ee59b1e0646a69722d0debff5\": container with ID starting with 3b51a2e3313b390ba71d9224ab277bbb3367033ee59b1e0646a69722d0debff5 not found: ID does not exist" containerID="3b51a2e3313b390ba71d9224ab277bbb3367033ee59b1e0646a69722d0debff5"
Oct 30 13:42:30.521982 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.521952    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b51a2e3313b390ba71d9224ab277bbb3367033ee59b1e0646a69722d0debff5"} err="failed to get container status \"3b51a2e3313b390ba71d9224ab277bbb3367033ee59b1e0646a69722d0debff5\": rpc error: code = NotFound desc = could not find container \"3b51a2e3313b390ba71d9224ab277bbb3367033ee59b1e0646a69722d0debff5\": container with ID starting with 3b51a2e3313b390ba71d9224ab277bbb3367033ee59b1e0646a69722d0debff5 not found: ID does not exist"
Oct 30 13:42:30.548627 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.548581    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"]
Oct 30 13:42:30.558323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:30.558296    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-51288b-openshift-default-7dc57d6bdc-v62kx"]
Oct 30 13:42:32.529356 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:32.529325    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" path="/var/lib/kubelet/pods/daffe0a0-8e1d-4bbf-a044-67a5f3af103a/volumes"
Oct 30 13:42:38.864101 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:38.864066    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"]
Oct 30 13:42:38.865012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:38.864283    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n" podUID="4f7d110c-715f-4265-9e9f-542bfb750b27" containerName="maas-api" containerID="cri-o://56edf7b4ea7669a22d9531f07482de471fec6f4d36142f1ee7359d63db25c69d" gracePeriod=30
Oct 30 13:42:39.123927 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.123861    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:42:39.173553 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.173521    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-maas-api-tls\") pod \"4f7d110c-715f-4265-9e9f-542bfb750b27\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") "
Oct 30 13:42:39.173755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.173574    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-metrics-certs\") pod \"4f7d110c-715f-4265-9e9f-542bfb750b27\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") "
Oct 30 13:42:39.173755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.173641    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f7d110c-715f-4265-9e9f-542bfb750b27-kube-api-access-rtjqx\" (UniqueName: \"kubernetes.io/projected/4f7d110c-715f-4265-9e9f-542bfb750b27-kube-api-access-rtjqx\") pod \"4f7d110c-715f-4265-9e9f-542bfb750b27\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") "
Oct 30 13:42:39.173755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.173672    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f7d110c-715f-4265-9e9f-542bfb750b27-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d110c-715f-4265-9e9f-542bfb750b27-openshift-service-ca\") pod \"4f7d110c-715f-4265-9e9f-542bfb750b27\" (UID: \"4f7d110c-715f-4265-9e9f-542bfb750b27\") "
Oct 30 13:42:39.174742 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.174705    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f7d110c-715f-4265-9e9f-542bfb750b27-openshift-service-ca" pod "4f7d110c-715f-4265-9e9f-542bfb750b27" (UID: "4f7d110c-715f-4265-9e9f-542bfb750b27"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:39.176225 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.176187    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-metrics-certs" pod "4f7d110c-715f-4265-9e9f-542bfb750b27" (UID: "4f7d110c-715f-4265-9e9f-542bfb750b27"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:39.176430 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.176360    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-maas-api-tls" pod "4f7d110c-715f-4265-9e9f-542bfb750b27" (UID: "4f7d110c-715f-4265-9e9f-542bfb750b27"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:39.176632 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.176609    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f7d110c-715f-4265-9e9f-542bfb750b27-kube-api-access-rtjqx" pod "4f7d110c-715f-4265-9e9f-542bfb750b27" (UID: "4f7d110c-715f-4265-9e9f-542bfb750b27"). InnerVolumeSpecName "kube-api-access-rtjqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:39.275676 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.275639    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:39.275676 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.275669    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4f7d110c-715f-4265-9e9f-542bfb750b27-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:39.275676 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.275679    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtjqx\" (UniqueName: \"kubernetes.io/projected/4f7d110c-715f-4265-9e9f-542bfb750b27-kube-api-access-rtjqx\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:39.275992 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.275689    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4f7d110c-715f-4265-9e9f-542bfb750b27-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:39.556663 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.556625    5893 generic.go:356] "Generic (PLEG): container finished" podID="4f7d110c-715f-4265-9e9f-542bfb750b27" containerID="56edf7b4ea7669a22d9531f07482de471fec6f4d36142f1ee7359d63db25c69d" exitCode=0
Oct 30 13:42:39.556886 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.556685    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"
Oct 30 13:42:39.556886 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.556734    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n" event={"ID":"4f7d110c-715f-4265-9e9f-542bfb750b27","Type":"ContainerDied","Data":"56edf7b4ea7669a22d9531f07482de471fec6f4d36142f1ee7359d63db25c69d"}
Oct 30 13:42:39.556886 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.556816    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n" event={"ID":"4f7d110c-715f-4265-9e9f-542bfb750b27","Type":"ContainerDied","Data":"62cb5f96e79ca3e027b638837be1906b4d2f39ac669d31d49bd7e10e0ea48ec4"}
Oct 30 13:42:39.556886 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.556847    5893 scope.go:122] "RemoveContainer" containerID="56edf7b4ea7669a22d9531f07482de471fec6f4d36142f1ee7359d63db25c69d"
Oct 30 13:42:39.568685 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.568661    5893 scope.go:122] "RemoveContainer" containerID="56edf7b4ea7669a22d9531f07482de471fec6f4d36142f1ee7359d63db25c69d"
Oct 30 13:42:39.568993 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:42:39.568969    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56edf7b4ea7669a22d9531f07482de471fec6f4d36142f1ee7359d63db25c69d\": container with ID starting with 56edf7b4ea7669a22d9531f07482de471fec6f4d36142f1ee7359d63db25c69d not found: ID does not exist" containerID="56edf7b4ea7669a22d9531f07482de471fec6f4d36142f1ee7359d63db25c69d"
Oct 30 13:42:39.569076 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.569007    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56edf7b4ea7669a22d9531f07482de471fec6f4d36142f1ee7359d63db25c69d"} err="failed to get container status \"56edf7b4ea7669a22d9531f07482de471fec6f4d36142f1ee7359d63db25c69d\": rpc error: code = NotFound desc = could not find container \"56edf7b4ea7669a22d9531f07482de471fec6f4d36142f1ee7359d63db25c69d\": container with ID starting with 56edf7b4ea7669a22d9531f07482de471fec6f4d36142f1ee7359d63db25c69d not found: ID does not exist"
Oct 30 13:42:39.609549 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.609514    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"]
Oct 30 13:42:39.618088 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:39.618054    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-627dfa-68574958fd-pkf2n"]
Oct 30 13:42:40.530234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:40.530203    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f7d110c-715f-4265-9e9f-542bfb750b27" path="/var/lib/kubelet/pods/4f7d110c-715f-4265-9e9f-542bfb750b27/volumes"
Oct 30 13:42:47.496816 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:47.496783    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"]
Oct 30 13:42:47.497293 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:47.497073    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj" podUID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerName="istio-proxy" containerID="cri-o://7572db77ec5f5ecb7ac4fc3cc6dadf4bf39dab082a747ca3e83513b88d4cd8a5" gracePeriod=30
Oct 30 13:42:48.644780 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:42:48.644742    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:42:52.632407 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.632372    5893 generic.go:356] "Generic (PLEG): container finished" podID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerID="7572db77ec5f5ecb7ac4fc3cc6dadf4bf39dab082a747ca3e83513b88d4cd8a5" exitCode=0
Oct 30 13:42:52.632838 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.632461    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj" event={"ID":"29e1be4b-6574-45e6-b865-9159a3526f2f","Type":"ContainerDied","Data":"7572db77ec5f5ecb7ac4fc3cc6dadf4bf39dab082a747ca3e83513b88d4cd8a5"}
Oct 30 13:42:52.632838 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.632571    5893 scope.go:122] "RemoveContainer" containerID="be35fa895f7b318db1978454e1afca757f43b2008b2e20067e19c013d2a34efe"
Oct 30 13:42:52.756980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.756952    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:42:52.914242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.914149    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-podinfo\") pod \"29e1be4b-6574-45e6-b865-9159a3526f2f\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") "
Oct 30 13:42:52.914242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.914200    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e1be4b-6574-45e6-b865-9159a3526f2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e1be4b-6574-45e6-b865-9159a3526f2f-istiod-ca-cert\") pod \"29e1be4b-6574-45e6-b865-9159a3526f2f\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") "
Oct 30 13:42:52.914242 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.914237    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-workload-socket\") pod \"29e1be4b-6574-45e6-b865-9159a3526f2f\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") "
Oct 30 13:42:52.914579 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.914307    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e1be4b-6574-45e6-b865-9159a3526f2f-kube-api-access-76sxf\" (UniqueName: \"kubernetes.io/projected/29e1be4b-6574-45e6-b865-9159a3526f2f-kube-api-access-76sxf\") pod \"29e1be4b-6574-45e6-b865-9159a3526f2f\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") "
Oct 30 13:42:52.914579 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.914376    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-token\" (UniqueName: \"kubernetes.io/projected/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-token\") pod \"29e1be4b-6574-45e6-b865-9159a3526f2f\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") "
Oct 30 13:42:52.914579 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.914410    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-credential-socket\") pod \"29e1be4b-6574-45e6-b865-9159a3526f2f\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") "
Oct 30 13:42:52.914579 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.914481    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-envoy\") pod \"29e1be4b-6574-45e6-b865-9159a3526f2f\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") "
Oct 30 13:42:52.914579 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.914514    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-data\") pod \"29e1be4b-6574-45e6-b865-9159a3526f2f\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") "
Oct 30 13:42:52.914579 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.914546    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-workload-certs\") pod \"29e1be4b-6574-45e6-b865-9159a3526f2f\" (UID: \"29e1be4b-6574-45e6-b865-9159a3526f2f\") "
Oct 30 13:42:52.914907 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.914773    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e1be4b-6574-45e6-b865-9159a3526f2f-istiod-ca-cert" pod "29e1be4b-6574-45e6-b865-9159a3526f2f" (UID: "29e1be4b-6574-45e6-b865-9159a3526f2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:52.914968 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.914916    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29e1be4b-6574-45e6-b865-9159a3526f2f-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:52.915111 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.915091    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-data" pod "29e1be4b-6574-45e6-b865-9159a3526f2f" (UID: "29e1be4b-6574-45e6-b865-9159a3526f2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:52.915383 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.915363    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-workload-socket" pod "29e1be4b-6574-45e6-b865-9159a3526f2f" (UID: "29e1be4b-6574-45e6-b865-9159a3526f2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:52.915467 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.915422    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-credential-socket" pod "29e1be4b-6574-45e6-b865-9159a3526f2f" (UID: "29e1be4b-6574-45e6-b865-9159a3526f2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:52.915744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.915682    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-workload-certs" pod "29e1be4b-6574-45e6-b865-9159a3526f2f" (UID: "29e1be4b-6574-45e6-b865-9159a3526f2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:52.917012 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.916987    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-podinfo" pod "29e1be4b-6574-45e6-b865-9159a3526f2f" (UID: "29e1be4b-6574-45e6-b865-9159a3526f2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:42:52.917387 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.917357    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-token" pod "29e1be4b-6574-45e6-b865-9159a3526f2f" (UID: "29e1be4b-6574-45e6-b865-9159a3526f2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:52.917473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.917377    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-envoy" pod "29e1be4b-6574-45e6-b865-9159a3526f2f" (UID: "29e1be4b-6574-45e6-b865-9159a3526f2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:42:52.917473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:52.917390    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e1be4b-6574-45e6-b865-9159a3526f2f-kube-api-access-76sxf" pod "29e1be4b-6574-45e6-b865-9159a3526f2f" (UID: "29e1be4b-6574-45e6-b865-9159a3526f2f"). InnerVolumeSpecName "kube-api-access-76sxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:53.006712 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:53.006674    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"]
Oct 30 13:42:53.016048 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:53.016015    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:53.016048 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:53.016048    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:53.016240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:53.016060    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:53.016240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:53.016074    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:53.016240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:53.016085    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:53.016240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:53.016099    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76sxf\" (UniqueName: \"kubernetes.io/projected/29e1be4b-6574-45e6-b865-9159a3526f2f-kube-api-access-76sxf\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:53.016240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:53.016112    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29e1be4b-6574-45e6-b865-9159a3526f2f-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:53.016240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:53.016124    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29e1be4b-6574-45e6-b865-9159a3526f2f-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:42:53.638925 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:53.638890    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj" event={"ID":"29e1be4b-6574-45e6-b865-9159a3526f2f","Type":"ContainerDied","Data":"9f8a72fb89ee28f3b949675fa0b0db643c3602fe5ad1c451b22a9ebd84fb9df9"}
Oct 30 13:42:53.638925 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:53.638913    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"
Oct 30 13:42:53.639468 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:53.638951    5893 scope.go:122] "RemoveContainer" containerID="7572db77ec5f5ecb7ac4fc3cc6dadf4bf39dab082a747ca3e83513b88d4cd8a5"
Oct 30 13:42:53.689760 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:53.689725    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"]
Oct 30 13:42:53.699829 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:53.699801    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-627dfa-openshift-default-6d659cfb85-6brnj"]
Oct 30 13:42:54.530737 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:54.530702    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e1be4b-6574-45e6-b865-9159a3526f2f" path="/var/lib/kubelet/pods/29e1be4b-6574-45e6-b865-9159a3526f2f/volumes"
Oct 30 13:42:59.850720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.850684    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"]
Oct 30 13:42:59.851132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851067    5893 cpu_manager.go:414] "containerMap: removing container" podUID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerName="istio-proxy"
Oct 30 13:42:59.851132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851078    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerName="istio-proxy"
Oct 30 13:42:59.851132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851088    5893 cpu_manager.go:414] "containerMap: removing container" podUID="fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9" containerName="payload-processing"
Oct 30 13:42:59.851132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851093    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9" containerName="payload-processing"
Oct 30 13:42:59.851132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851117    5893 cpu_manager.go:414] "containerMap: removing container" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerName="istio-proxy"
Oct 30 13:42:59.851132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851123    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerName="istio-proxy"
Oct 30 13:42:59.851132 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851136    5893 cpu_manager.go:414] "containerMap: removing container" podUID="47ecad36-94c6-4bf6-8738-a5c19fe9ba78" containerName="maas-api"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851142    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47ecad36-94c6-4bf6-8738-a5c19fe9ba78" containerName="maas-api"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851148    5893 cpu_manager.go:414] "containerMap: removing container" podUID="aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" containerName="istio-proxy"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851153    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" containerName="istio-proxy"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851159    5893 cpu_manager.go:414] "containerMap: removing container" podUID="c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" containerName="istio-proxy"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851163    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" containerName="istio-proxy"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851170    5893 cpu_manager.go:414] "containerMap: removing container" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerName="istio-proxy"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851174    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerName="istio-proxy"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851180    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d84ed3c0-d5fc-4961-a037-1c20a8704e50" containerName="maas-api"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851184    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d84ed3c0-d5fc-4961-a037-1c20a8704e50" containerName="maas-api"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851190    5893 cpu_manager.go:414] "containerMap: removing container" podUID="f02e2238-4a09-4cf3-b20a-d82b0aba4fe3" containerName="payload-pre-processing"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851194    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f02e2238-4a09-4cf3-b20a-d82b0aba4fe3" containerName="payload-pre-processing"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851201    5893 cpu_manager.go:414] "containerMap: removing container" podUID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerName="istio-proxy"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851204    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerName="istio-proxy"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851222    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d71220ae-508e-46fa-8093-fcd9c9bb3231" containerName="payload-pre-processing"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851226    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71220ae-508e-46fa-8093-fcd9c9bb3231" containerName="payload-pre-processing"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851235    5893 cpu_manager.go:414] "containerMap: removing container" podUID="4f7d110c-715f-4265-9e9f-542bfb750b27" containerName="maas-api"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851240    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f7d110c-715f-4265-9e9f-542bfb750b27" containerName="maas-api"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851309    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f7d110c-715f-4265-9e9f-542bfb750b27" containerName="maas-api"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851321    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerName="istio-proxy"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851332    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce9f69f-12ca-4fc7-84eb-d2c1dae96fb9" containerName="payload-processing"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851342    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerName="istio-proxy"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851349    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="47ecad36-94c6-4bf6-8738-a5c19fe9ba78" containerName="maas-api"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851357    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac0cb89-f88f-4fe8-a9fa-8080c0d9f6fe" containerName="istio-proxy"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851364    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="c80dd06d-424b-4ab3-ba9d-deb0d6ac166d" containerName="istio-proxy"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851371    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e02b51d-4bfe-4805-b52e-d65419291f1e" containerName="istio-proxy"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851380    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d84ed3c0-d5fc-4961-a037-1c20a8704e50" containerName="maas-api"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851390    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="f02e2238-4a09-4cf3-b20a-d82b0aba4fe3" containerName="payload-pre-processing"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851397    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerName="istio-proxy"
Oct 30 13:42:59.851451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.851408    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71220ae-508e-46fa-8093-fcd9c9bb3231" containerName="payload-pre-processing"
Oct 30 13:42:59.854411 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.854396    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"
Oct 30 13:42:59.866729 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.866704    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:42:59.876829 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.876800    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7d9m2\""
Oct 30 13:42:59.878441 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.878415    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"]
Oct 30 13:42:59.985980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.985942    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f454da5-2e40-41af-b831-7742a07cb899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-686264ec-tghsr\" (UID: \"7f454da5-2e40-41af-b831-7742a07cb899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"
Oct 30 13:42:59.986175 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.985993    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f454da5-2e40-41af-b831-7742a07cb899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-686264ec-tghsr\" (UID: \"7f454da5-2e40-41af-b831-7742a07cb899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"
Oct 30 13:42:59.986175 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:42:59.986012    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9h7q\" (UniqueName: \"kubernetes.io/projected/7f454da5-2e40-41af-b831-7742a07cb899-kube-api-access-j9h7q\") pod \"maas-api-revoke-sub-686264ec-tghsr\" (UID: \"7f454da5-2e40-41af-b831-7742a07cb899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"
Oct 30 13:43:00.087021 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:00.086987    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f454da5-2e40-41af-b831-7742a07cb899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-686264ec-tghsr\" (UID: \"7f454da5-2e40-41af-b831-7742a07cb899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"
Oct 30 13:43:00.087203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:00.087039    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f454da5-2e40-41af-b831-7742a07cb899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-686264ec-tghsr\" (UID: \"7f454da5-2e40-41af-b831-7742a07cb899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"
Oct 30 13:43:00.087203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:00.087069    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9h7q\" (UniqueName: \"kubernetes.io/projected/7f454da5-2e40-41af-b831-7742a07cb899-kube-api-access-j9h7q\") pod \"maas-api-revoke-sub-686264ec-tghsr\" (UID: \"7f454da5-2e40-41af-b831-7742a07cb899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"
Oct 30 13:43:00.087891 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:00.087844    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f454da5-2e40-41af-b831-7742a07cb899-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-686264ec-tghsr\" (UID: \"7f454da5-2e40-41af-b831-7742a07cb899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"
Oct 30 13:43:00.089596 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:00.089569    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f454da5-2e40-41af-b831-7742a07cb899-maas-api-xks-ca\") pod \"maas-api-revoke-sub-686264ec-tghsr\" (UID: \"7f454da5-2e40-41af-b831-7742a07cb899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"
Oct 30 13:43:00.108924 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:00.108855    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9h7q\" (UniqueName: \"kubernetes.io/projected/7f454da5-2e40-41af-b831-7742a07cb899-kube-api-access-j9h7q\") pod \"maas-api-revoke-sub-686264ec-tghsr\" (UID: \"7f454da5-2e40-41af-b831-7742a07cb899\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"
Oct 30 13:43:00.167658 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:00.167620    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"
Oct 30 13:43:00.472423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:00.472393    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"]
Oct 30 13:43:00.478866 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:43:00.478835    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f454da5_2e40_41af_b831_7742a07cb899.slice/crio-8752995ed102d1231bbc3b89d899c7d3fd3907afba149f36a074a4f372ac1ca7 WatchSource:0}: Error finding container 8752995ed102d1231bbc3b89d899c7d3fd3907afba149f36a074a4f372ac1ca7: Status 404 returned error can't find the container with id 8752995ed102d1231bbc3b89d899c7d3fd3907afba149f36a074a4f372ac1ca7
Oct 30 13:43:00.677559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:00.677530    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr" event={"ID":"7f454da5-2e40-41af-b831-7742a07cb899","Type":"ContainerStarted","Data":"8752995ed102d1231bbc3b89d899c7d3fd3907afba149f36a074a4f372ac1ca7"}
Oct 30 13:43:01.684806 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:01.684719    5893 generic.go:356] "Generic (PLEG): container finished" podID="7f454da5-2e40-41af-b831-7742a07cb899" containerID="7d4eaa450282a72d0a7505c311c91838b83e18dd31a10641b0ecf7d9e3bde631" exitCode=0
Oct 30 13:43:01.685226 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:01.684819    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr" event={"ID":"7f454da5-2e40-41af-b831-7742a07cb899","Type":"ContainerDied","Data":"7d4eaa450282a72d0a7505c311c91838b83e18dd31a10641b0ecf7d9e3bde631"}
Oct 30 13:43:02.821777 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:02.821746    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"
Oct 30 13:43:02.914788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:02.914756    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f454da5-2e40-41af-b831-7742a07cb899-kube-api-access-j9h7q\" (UniqueName: \"kubernetes.io/projected/7f454da5-2e40-41af-b831-7742a07cb899-kube-api-access-j9h7q\") pod \"7f454da5-2e40-41af-b831-7742a07cb899\" (UID: \"7f454da5-2e40-41af-b831-7742a07cb899\") "
Oct 30 13:43:02.914965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:02.914872    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f454da5-2e40-41af-b831-7742a07cb899-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f454da5-2e40-41af-b831-7742a07cb899-maas-api-openshift-ca\") pod \"7f454da5-2e40-41af-b831-7742a07cb899\" (UID: \"7f454da5-2e40-41af-b831-7742a07cb899\") "
Oct 30 13:43:02.914965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:02.914900    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f454da5-2e40-41af-b831-7742a07cb899-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f454da5-2e40-41af-b831-7742a07cb899-maas-api-xks-ca\") pod \"7f454da5-2e40-41af-b831-7742a07cb899\" (UID: \"7f454da5-2e40-41af-b831-7742a07cb899\") "
Oct 30 13:43:02.915235 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:02.915213    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f454da5-2e40-41af-b831-7742a07cb899-maas-api-openshift-ca" pod "7f454da5-2e40-41af-b831-7742a07cb899" (UID: "7f454da5-2e40-41af-b831-7742a07cb899"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:02.917130 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:02.917096    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f454da5-2e40-41af-b831-7742a07cb899-kube-api-access-j9h7q" pod "7f454da5-2e40-41af-b831-7742a07cb899" (UID: "7f454da5-2e40-41af-b831-7742a07cb899"). InnerVolumeSpecName "kube-api-access-j9h7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:02.917244 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:02.917101    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f454da5-2e40-41af-b831-7742a07cb899-maas-api-xks-ca" pod "7f454da5-2e40-41af-b831-7742a07cb899" (UID: "7f454da5-2e40-41af-b831-7742a07cb899"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:03.016390 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:03.016356    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f454da5-2e40-41af-b831-7742a07cb899-maas-api-openshift-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:43:03.016390 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:03.016384    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f454da5-2e40-41af-b831-7742a07cb899-maas-api-xks-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:43:03.016390 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:03.016394    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9h7q\" (UniqueName: \"kubernetes.io/projected/7f454da5-2e40-41af-b831-7742a07cb899-kube-api-access-j9h7q\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:43:03.695673 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:03.695641    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"
Oct 30 13:43:03.695673 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:03.695662    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr" event={"ID":"7f454da5-2e40-41af-b831-7742a07cb899","Type":"ContainerDied","Data":"8752995ed102d1231bbc3b89d899c7d3fd3907afba149f36a074a4f372ac1ca7"}
Oct 30 13:43:03.695901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:03.695712    5893 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8752995ed102d1231bbc3b89d899c7d3fd3907afba149f36a074a4f372ac1ca7"
Oct 30 13:43:08.027892 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.027833    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" containerID="cri-o://3df6f58f7d7c5b76d2b1b7dd355ad246856cbdb0df1502b0b4492abcac7f0633" gracePeriod=45
Oct 30 13:43:08.109211 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.109176    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": dial tcp 10.134.0.61:8000: connect: connection refused"
Oct 30 13:43:08.277485 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.277459    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:43:08.370495 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.370404    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-dshm\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-dshm\") pod \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") "
Oct 30 13:43:08.370703 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.370535    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-model-cache\") pod \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") "
Oct 30 13:43:08.370703 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.370599    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-home\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-home\") pod \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") "
Oct 30 13:43:08.370703 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.370637    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-tmp-dir\") pod \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") "
Oct 30 13:43:08.370703 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.370697    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/514b6aca-9ccc-4f7e-833b-1a5d32969e37-kube-api-access-nl7nr\" (UniqueName: \"kubernetes.io/projected/514b6aca-9ccc-4f7e-833b-1a5d32969e37-kube-api-access-nl7nr\") pod \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") "
Oct 30 13:43:08.371113 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.370746    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/514b6aca-9ccc-4f7e-833b-1a5d32969e37-tls-certs\" (UniqueName: \"kubernetes.io/secret/514b6aca-9ccc-4f7e-833b-1a5d32969e37-tls-certs\") pod \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\" (UID: \"514b6aca-9ccc-4f7e-833b-1a5d32969e37\") "
Oct 30 13:43:08.371113 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.370859    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-model-cache" pod "514b6aca-9ccc-4f7e-833b-1a5d32969e37" (UID: "514b6aca-9ccc-4f7e-833b-1a5d32969e37"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:08.371113 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.370887    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-home" pod "514b6aca-9ccc-4f7e-833b-1a5d32969e37" (UID: "514b6aca-9ccc-4f7e-833b-1a5d32969e37"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:08.371113 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.370938    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-tmp-dir" pod "514b6aca-9ccc-4f7e-833b-1a5d32969e37" (UID: "514b6aca-9ccc-4f7e-833b-1a5d32969e37"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:08.371436 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.371263    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:43:08.371436 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.371312    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:43:08.371436 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.371325    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:43:08.373622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.373596    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/514b6aca-9ccc-4f7e-833b-1a5d32969e37-kube-api-access-nl7nr" pod "514b6aca-9ccc-4f7e-833b-1a5d32969e37" (UID: "514b6aca-9ccc-4f7e-833b-1a5d32969e37"). InnerVolumeSpecName "kube-api-access-nl7nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:08.373622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.373610    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/514b6aca-9ccc-4f7e-833b-1a5d32969e37-tls-certs" pod "514b6aca-9ccc-4f7e-833b-1a5d32969e37" (UID: "514b6aca-9ccc-4f7e-833b-1a5d32969e37"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:08.373622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.373602    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-dshm" pod "514b6aca-9ccc-4f7e-833b-1a5d32969e37" (UID: "514b6aca-9ccc-4f7e-833b-1a5d32969e37"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:43:08.472796 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.472763    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl7nr\" (UniqueName: \"kubernetes.io/projected/514b6aca-9ccc-4f7e-833b-1a5d32969e37-kube-api-access-nl7nr\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:43:08.472796 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.472795    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/514b6aca-9ccc-4f7e-833b-1a5d32969e37-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:43:08.472995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.472808    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/514b6aca-9ccc-4f7e-833b-1a5d32969e37-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:43:08.720037 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.720010    5893 generic.go:356] "Generic (PLEG): container finished" podID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerID="3df6f58f7d7c5b76d2b1b7dd355ad246856cbdb0df1502b0b4492abcac7f0633" exitCode=0
Oct 30 13:43:08.720215 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.720054    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"
Oct 30 13:43:08.720215 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.720094    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" event={"ID":"514b6aca-9ccc-4f7e-833b-1a5d32969e37","Type":"ContainerDied","Data":"3df6f58f7d7c5b76d2b1b7dd355ad246856cbdb0df1502b0b4492abcac7f0633"}
Oct 30 13:43:08.720215 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.720144    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg" event={"ID":"514b6aca-9ccc-4f7e-833b-1a5d32969e37","Type":"ContainerDied","Data":"d2111039b2c09a88fc86dd4ecd858b834b84e361536c8e59033bf912ee2d649e"}
Oct 30 13:43:08.720215 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.720163    5893 scope.go:122] "RemoveContainer" containerID="3df6f58f7d7c5b76d2b1b7dd355ad246856cbdb0df1502b0b4492abcac7f0633"
Oct 30 13:43:08.729422 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.729396    5893 scope.go:122] "RemoveContainer" containerID="3df6f58f7d7c5b76d2b1b7dd355ad246856cbdb0df1502b0b4492abcac7f0633"
Oct 30 13:43:08.729672 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:43:08.729655    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3df6f58f7d7c5b76d2b1b7dd355ad246856cbdb0df1502b0b4492abcac7f0633\": container with ID starting with 3df6f58f7d7c5b76d2b1b7dd355ad246856cbdb0df1502b0b4492abcac7f0633 not found: ID does not exist" containerID="3df6f58f7d7c5b76d2b1b7dd355ad246856cbdb0df1502b0b4492abcac7f0633"
Oct 30 13:43:08.729733 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.729680    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3df6f58f7d7c5b76d2b1b7dd355ad246856cbdb0df1502b0b4492abcac7f0633"} err="failed to get container status \"3df6f58f7d7c5b76d2b1b7dd355ad246856cbdb0df1502b0b4492abcac7f0633\": rpc error: code = NotFound desc = could not find container \"3df6f58f7d7c5b76d2b1b7dd355ad246856cbdb0df1502b0b4492abcac7f0633\": container with ID starting with 3df6f58f7d7c5b76d2b1b7dd355ad246856cbdb0df1502b0b4492abcac7f0633 not found: ID does not exist"
Oct 30 13:43:08.765464 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.765429    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"]
Oct 30 13:43:08.775602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:08.775570    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-67f56e/distinct-2-w4-67f56e-kserve-64c7cf5897-vc6dg"]
Oct 30 13:43:10.530576 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:10.530539    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" path="/var/lib/kubelet/pods/514b6aca-9ccc-4f7e-833b-1a5d32969e37/volumes"
Oct 30 13:43:18.645265 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:43:18.645227    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:43:48.644831 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:43:48.644784    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:43:48.723198 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.723164    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"]
Oct 30 13:43:48.723727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.723705    5893 cpu_manager.go:414] "containerMap: removing container" podUID="7f454da5-2e40-41af-b831-7742a07cb899" containerName="revoke-keys"
Oct 30 13:43:48.723727 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.723724    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f454da5-2e40-41af-b831-7742a07cb899" containerName="revoke-keys"
Oct 30 13:43:48.723900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.723739    5893 cpu_manager.go:414] "containerMap: removing container" podUID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerName="istio-proxy"
Oct 30 13:43:48.723900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.723748    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerName="istio-proxy"
Oct 30 13:43:48.723900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.723771    5893 cpu_manager.go:414] "containerMap: removing container" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main"
Oct 30 13:43:48.723900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.723776    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main"
Oct 30 13:43:48.723900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.723793    5893 cpu_manager.go:414] "containerMap: removing container" podUID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerName="istio-proxy"
Oct 30 13:43:48.723900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.723798    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerName="istio-proxy"
Oct 30 13:43:48.723900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.723894    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f454da5-2e40-41af-b831-7742a07cb899" containerName="revoke-keys"
Oct 30 13:43:48.723900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.723901    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="daffe0a0-8e1d-4bbf-a044-67a5f3af103a" containerName="istio-proxy"
Oct 30 13:43:48.724149 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.723914    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="514b6aca-9ccc-4f7e-833b-1a5d32969e37" containerName="main"
Oct 30 13:43:48.724149 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.723927    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e1be4b-6574-45e6-b865-9159a3526f2f" containerName="istio-proxy"
Oct 30 13:43:48.726591 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.726574    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.752210 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.752187    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-4s79t\""
Oct 30 13:43:48.758793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.758769    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 13:43:48.761222 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.761198    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.761316 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.761239    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.761385 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.761328    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.761443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.761404    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.761500 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.761450    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.761500 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.761487    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.761610 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.761511    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.761610 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.761533    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2xw2\" (UniqueName: \"kubernetes.io/projected/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-kube-api-access-p2xw2\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.761773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.761618    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.791841 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.791807    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"]
Oct 30 13:43:48.862408 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.862374    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.862594 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.862421    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.862594 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.862437    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.862594 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.862453    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2xw2\" (UniqueName: \"kubernetes.io/projected/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-kube-api-access-p2xw2\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.862594 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.862472    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.862594 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.862518    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.862594 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.862559    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.862938 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.862621    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.862938 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.862654    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.862938 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.862867    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.863107 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.862948    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.863107 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.863040    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.863229 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.863198    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.863316 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.863223    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.864966 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.864947    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.865208 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.865181    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.884688 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.884664    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:48.884797 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:48.884755    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2xw2\" (UniqueName: \"kubernetes.io/projected/c66344b3-3175-40c0-b8a3-51ccf36bdc4f-kube-api-access-p2xw2\") pod \"maas-default-gateway-openshift-default-58c6585447-ktgn6\" (UID: \"c66344b3-3175-40c0-b8a3-51ccf36bdc4f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:49.039211 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:49.039177    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:49.359686 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:49.359605    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"]
Oct 30 13:43:49.363216 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:43:49.363190    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66344b3_3175_40c0_b8a3_51ccf36bdc4f.slice/crio-7d20d4469b1dfe1349648377b5eea90e25b1186c93bed4bffc393d5a3afe4905 WatchSource:0}: Error finding container 7d20d4469b1dfe1349648377b5eea90e25b1186c93bed4bffc393d5a3afe4905: Status 404 returned error can't find the container with id 7d20d4469b1dfe1349648377b5eea90e25b1186c93bed4bffc393d5a3afe4905
Oct 30 13:43:49.365309 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:49.365260    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:43:49.365428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:49.365356    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:43:49.365428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:49.365400    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:43:49.928705 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:49.928663    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6" event={"ID":"c66344b3-3175-40c0-b8a3-51ccf36bdc4f","Type":"ContainerStarted","Data":"2b0e0358fba7dc150c7656221fc4c051a0d02ba735013e2bda1982419ddeaabe"}
Oct 30 13:43:49.929111 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:49.928751    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6" event={"ID":"c66344b3-3175-40c0-b8a3-51ccf36bdc4f","Type":"ContainerStarted","Data":"7d20d4469b1dfe1349648377b5eea90e25b1186c93bed4bffc393d5a3afe4905"}
Oct 30 13:43:49.995553 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:49.995496    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6" podStartSLOduration=1.9954815689999998 podStartE2EDuration="1.995481569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:49.985532917 +0000 UTC m=+3471.568962623" watchObservedRunningTime="2026-10-30 13:43:49.995481569 +0000 UTC m=+3471.578911262"
Oct 30 13:43:50.040164 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:50.040131    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:50.062297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:50.058428    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 13:43:50.062297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:50.058517    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6" podUID="c66344b3-3175-40c0-b8a3-51ccf36bdc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 13:43:51.039815 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:51.039784    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 13:43:51.040222 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:51.039850    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6" podUID="c66344b3-3175-40c0-b8a3-51ccf36bdc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 13:43:52.065492 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:52.065436    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 13:43:52.066021 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:52.065519    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6" podUID="c66344b3-3175-40c0-b8a3-51ccf36bdc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 13:43:53.039981 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:53.039946    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 13:43:53.040147 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:53.040006    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6" podUID="c66344b3-3175-40c0-b8a3-51ccf36bdc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 13:43:54.058299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:54.040421    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 13:43:54.058299 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:54.040482    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6" podUID="c66344b3-3175-40c0-b8a3-51ccf36bdc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 13:43:55.039829 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:55.039793    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 13:43:55.040023 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:55.039855    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6" podUID="c66344b3-3175-40c0-b8a3-51ccf36bdc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 13:43:56.040374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:56.040332    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Oct 30 13:43:56.040780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:56.040403    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6" podUID="c66344b3-3175-40c0-b8a3-51ccf36bdc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Oct 30 13:43:58.039834 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:58.039781    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:43:58.040410 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:58.039851    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6" podUID="c66344b3-3175-40c0-b8a3-51ccf36bdc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:43:59.080473 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:59.080428    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ktgn6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:43:59.080999 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:59.080515    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6" podUID="c66344b3-3175-40c0-b8a3-51ccf36bdc4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:43:59.589842 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:59.589814    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:59.590120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:59.590094    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:43:59.590883 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:43:59.590865    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ktgn6"
Oct 30 13:44:18.644774 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:44:18.644728    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:44:48.645580 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:44:48.645541    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:45:03.912958 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:03.912919    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"]
Oct 30 13:45:03.917002 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:03.916980    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:03.951411 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:03.951380    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"]
Oct 30 13:45:04.003761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.003720    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f49c9da9-a8bc-4139-bcc9-617e5a360704-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.003957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.003820    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.003957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.003881    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f49c9da9-a8bc-4139-bcc9-617e5a360704-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.003957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.003944    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.004071 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.003963    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtx9w\" (UniqueName: \"kubernetes.io/projected/f49c9da9-a8bc-4139-bcc9-617e5a360704-kube-api-access-rtx9w\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.004071 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.003992    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.004142 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.004062    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.004142 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.004109    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.004216 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.004172    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f49c9da9-a8bc-4139-bcc9-617e5a360704-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.105230 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.105196    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.105429 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.105293    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f49c9da9-a8bc-4139-bcc9-617e5a360704-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.105429 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.105332    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f49c9da9-a8bc-4139-bcc9-617e5a360704-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.105429 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.105358    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.105429 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.105394    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f49c9da9-a8bc-4139-bcc9-617e5a360704-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.105650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.105440    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.105650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.105465    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtx9w\" (UniqueName: \"kubernetes.io/projected/f49c9da9-a8bc-4139-bcc9-617e5a360704-kube-api-access-rtx9w\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.105650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.105508    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.105650 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.105537    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.105890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.105688    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.105890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.105798    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.105890 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.105866    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.106060 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.105921    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.106152 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.106133    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f49c9da9-a8bc-4139-bcc9-617e5a360704-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.107922 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.107900    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f49c9da9-a8bc-4139-bcc9-617e5a360704-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.108035 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.108001    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f49c9da9-a8bc-4139-bcc9-617e5a360704-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.127775 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.127753    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f49c9da9-a8bc-4139-bcc9-617e5a360704-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.128127 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.128109    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtx9w\" (UniqueName: \"kubernetes.io/projected/f49c9da9-a8bc-4139-bcc9-617e5a360704-kube-api-access-rtx9w\") pod \"maas-default-gateway-openshift-default-58c6585447-ndplc\" (UID: \"f49c9da9-a8bc-4139-bcc9-617e5a360704\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.229659 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.229634    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:04.546345 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.546243    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"]
Oct 30 13:45:04.552003 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:45:04.551968    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf49c9da9_a8bc_4139_bcc9_617e5a360704.slice/crio-a472547881d1763879fbc442c533403e1e852e413ff35a7a98446ab1dbf05936 WatchSource:0}: Error finding container a472547881d1763879fbc442c533403e1e852e413ff35a7a98446ab1dbf05936: Status 404 returned error can't find the container with id a472547881d1763879fbc442c533403e1e852e413ff35a7a98446ab1dbf05936
Oct 30 13:45:04.554222 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.554188    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:45:04.554346 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.554328    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:45:04.554397 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:04.554367    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:45:05.310259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:05.310225    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc" event={"ID":"f49c9da9-a8bc-4139-bcc9-617e5a360704","Type":"ContainerStarted","Data":"b917fe0090349b1ea1173225a3bd90875b4d30b4a262322198a2defcfd499010"}
Oct 30 13:45:05.310688 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:05.310299    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc" event={"ID":"f49c9da9-a8bc-4139-bcc9-617e5a360704","Type":"ContainerStarted","Data":"a472547881d1763879fbc442c533403e1e852e413ff35a7a98446ab1dbf05936"}
Oct 30 13:45:05.366124 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:05.366071    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc" podStartSLOduration=2.366057353 podStartE2EDuration="2.366057353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:05.365849564 +0000 UTC m=+3546.949279279" watchObservedRunningTime="2026-10-30 13:45:05.366057353 +0000 UTC m=+3546.949487046"
Oct 30 13:45:06.230344 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:06.230305    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:06.231856 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:06.231830    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ndplc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:45:06.231994 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:06.231878    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc" podUID="f49c9da9-a8bc-4139-bcc9-617e5a360704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 13:45:07.230535 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:07.230494    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ndplc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:45:07.231006 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:07.230568    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc" podUID="f49c9da9-a8bc-4139-bcc9-617e5a360704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 13:45:08.231039 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:08.231004    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ndplc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:45:08.231465 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:08.231059    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc" podUID="f49c9da9-a8bc-4139-bcc9-617e5a360704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 13:45:09.230386 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:09.230352    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ndplc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:45:09.230559 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:09.230401    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc" podUID="f49c9da9-a8bc-4139-bcc9-617e5a360704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 13:45:10.230184 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:10.230144    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ndplc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:45:10.230636 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:10.230198    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc" podUID="f49c9da9-a8bc-4139-bcc9-617e5a360704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 13:45:11.230949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:11.230911    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-ndplc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Oct 30 13:45:11.231391 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:11.230975    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc" podUID="f49c9da9-a8bc-4139-bcc9-617e5a360704" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Oct 30 13:45:12.787152 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:12.787121    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:12.787649 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:12.787514    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:12.788638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:12.788616    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-ndplc"
Oct 30 13:45:18.662625 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:45:18.662547    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:45:18.976201 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:18.976169    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"]
Oct 30 13:45:18.980832 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:18.980805    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.014185 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.014140    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"]
Oct 30 13:45:19.109589 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.109551    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742ed464-ac90-468a-92f7-edde51ba02bb-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.109788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.109687    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.109788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.109746    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742ed464-ac90-468a-92f7-edde51ba02bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.109901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.109803    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742ed464-ac90-468a-92f7-edde51ba02bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.109901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.109825    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.109993 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.109903    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.109993 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.109934    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frwrk\" (UniqueName: \"kubernetes.io/projected/742ed464-ac90-468a-92f7-edde51ba02bb-kube-api-access-frwrk\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.109993 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.109961    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.110097 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.110050    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.211451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.211415    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.211652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.211458    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frwrk\" (UniqueName: \"kubernetes.io/projected/742ed464-ac90-468a-92f7-edde51ba02bb-kube-api-access-frwrk\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.211652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.211481    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.211652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.211534    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.211652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.211566    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742ed464-ac90-468a-92f7-edde51ba02bb-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.211652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.211621    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.212034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.211663    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742ed464-ac90-468a-92f7-edde51ba02bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.212034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.211690    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742ed464-ac90-468a-92f7-edde51ba02bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.212034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.211717    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.212034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.211896    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.212034 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.212004    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.212498 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.212466    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.212637 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.212493    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.212951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.212930    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/742ed464-ac90-468a-92f7-edde51ba02bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.213958 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.213935    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/742ed464-ac90-468a-92f7-edde51ba02bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.214611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.214587    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/742ed464-ac90-468a-92f7-edde51ba02bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.233326 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.233244    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/742ed464-ac90-468a-92f7-edde51ba02bb-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.233460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.233419    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frwrk\" (UniqueName: \"kubernetes.io/projected/742ed464-ac90-468a-92f7-edde51ba02bb-kube-api-access-frwrk\") pod \"maas-default-gateway-openshift-default-58c6585447-jl8n2\" (UID: \"742ed464-ac90-468a-92f7-edde51ba02bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.298188 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.298153    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:19.637658 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.637560    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"]
Oct 30 13:45:19.642862 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:45:19.642827    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742ed464_ac90_468a_92f7_edde51ba02bb.slice/crio-e08e281355d42dda2ed39b0f74bf9039a23bf3dd4df22cb80a67923a27478cdb WatchSource:0}: Error finding container e08e281355d42dda2ed39b0f74bf9039a23bf3dd4df22cb80a67923a27478cdb: Status 404 returned error can't find the container with id e08e281355d42dda2ed39b0f74bf9039a23bf3dd4df22cb80a67923a27478cdb
Oct 30 13:45:19.645684 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.645612    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:45:19.645780 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.645739    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:45:19.645821 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:19.645782    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:45:20.396056 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:20.396013    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2" event={"ID":"742ed464-ac90-468a-92f7-edde51ba02bb","Type":"ContainerStarted","Data":"35afc5c6895cfa74cc66052bd82a62624465891c3590d518b145dae4d3ab0e06"}
Oct 30 13:45:20.396486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:20.396122    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2" event={"ID":"742ed464-ac90-468a-92f7-edde51ba02bb","Type":"ContainerStarted","Data":"e08e281355d42dda2ed39b0f74bf9039a23bf3dd4df22cb80a67923a27478cdb"}
Oct 30 13:45:21.298957 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:21.298926    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:21.300569 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:21.300541    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-jl8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 30 13:45:21.300720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:21.300611    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2" podUID="742ed464-ac90-468a-92f7-edde51ba02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 30 13:45:22.299452 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:22.299418    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-jl8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 30 13:45:22.299840 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:22.299468    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2" podUID="742ed464-ac90-468a-92f7-edde51ba02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 30 13:45:23.299028 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:23.298990    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-jl8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 30 13:45:23.299234 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:23.299046    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2" podUID="742ed464-ac90-468a-92f7-edde51ba02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 30 13:45:24.299187 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:24.299152    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-jl8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 30 13:45:24.299652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:24.299205    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2" podUID="742ed464-ac90-468a-92f7-edde51ba02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 30 13:45:25.299256 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:25.299212    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-jl8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 30 13:45:25.299640 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:25.299293    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2" podUID="742ed464-ac90-468a-92f7-edde51ba02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 30 13:45:26.299089 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:26.299053    5893 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-jl8n2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Oct 30 13:45:26.299287 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:26.299119    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2" podUID="742ed464-ac90-468a-92f7-edde51ba02bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Oct 30 13:45:27.968497 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:27.968457    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:27.969016 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:27.968832    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:27.969662 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:27.969639    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2"
Oct 30 13:45:28.020649 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:45:28.020599    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-jl8n2" podStartSLOduration=10.020586783 podStartE2EDuration="10.020586783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:20.465587074 +0000 UTC m=+3562.049016795" watchObservedRunningTime="2026-10-30 13:45:28.020586783 +0000 UTC m=+3569.604016475"
Oct 30 13:45:48.644639 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:45:48.644599    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:46:01.439784 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:46:01.439754    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:46:01.452248 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:46:01.452223    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:46:01.458287 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:46:01.458247    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:46:01.469600 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:46:01.469579    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:46:18.644109 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:46:18.644057    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:46:48.645956 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:46:48.645919    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:47:18.644847 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:47:18.644803    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:47:48.644500 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:47:48.644415    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:48:03.037021 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:03.036980    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"]
Oct 30 13:48:03.038571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:03.038541    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-686264ec-tghsr"]
Oct 30 13:48:04.530854 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:04.530820    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f454da5-2e40-41af-b831-7742a07cb899" path="/var/lib/kubelet/pods/7f454da5-2e40-41af-b831-7742a07cb899/volumes"
Oct 30 13:48:18.644714 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:48:18.644673    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:48:18.951825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:18.951734    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"]
Oct 30 13:48:18.955720 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:18.955691    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:18.978332 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:18.978303    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-f53955-openshift-default-dockercfg-jnplz\""
Oct 30 13:48:18.990942 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:18.990908    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"]
Oct 30 13:48:19.052496 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.052464    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-workload-socket\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.052496 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.052500    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-workload-certs\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.052758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.052695    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-envoy\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.052758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.052740    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-data\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.052886 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.052787    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istiod-ca-cert\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.052886 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.052854    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-credential-socket\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.053011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.052922    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-podinfo\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.053011 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.052989    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-token\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.053119 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.053030    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf8kt\" (UniqueName: \"kubernetes.io/projected/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-kube-api-access-xf8kt\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.154609 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.154562    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-podinfo\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.154609 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.154603    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-token\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.154883 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.154629    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf8kt\" (UniqueName: \"kubernetes.io/projected/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-kube-api-access-xf8kt\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.154883 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.154686    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-workload-socket\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.154883 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.154705    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-workload-certs\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.154883 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.154801    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-envoy\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.154883 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.154835    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-data\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.154883 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.154875    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istiod-ca-cert\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.155231 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.154917    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-credential-socket\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.155364 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.155337    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-workload-certs\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.155446 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.155375    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-credential-socket\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.155654 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.155633    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-data\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.155746 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.155729    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istiod-ca-cert\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.156074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.156049    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-workload-socket\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.157495 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.157463    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-envoy\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.157750 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.157731    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-podinfo\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.176877 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.176832    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-token\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.177653 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.177622    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf8kt\" (UniqueName: \"kubernetes.io/projected/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-kube-api-access-xf8kt\") pod \"e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") " pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.269426 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.269393    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:19.603207 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.603127    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"]
Oct 30 13:48:19.608939 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:48:19.608915    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25b28ab_9fff_4fca_8416_ef79ecf1e78a.slice/crio-7634da023d0d7056ca5fc71dbc1159cbd3547588aa4167cf615c3a0077a3aab9 WatchSource:0}: Error finding container 7634da023d0d7056ca5fc71dbc1159cbd3547588aa4167cf615c3a0077a3aab9: Status 404 returned error can't find the container with id 7634da023d0d7056ca5fc71dbc1159cbd3547588aa4167cf615c3a0077a3aab9
Oct 30 13:48:19.611369 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.611333    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:48:19.611474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.611416    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:48:19.611474 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:19.611457    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:48:20.382618 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:20.382583    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq" event={"ID":"d25b28ab-9fff-4fca-8416-ef79ecf1e78a","Type":"ContainerStarted","Data":"9ba11186f32613d095202e52b628d530357423fc9e9d7d430734a761bc37ec11"}
Oct 30 13:48:20.383106 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:20.382633    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq" event={"ID":"d25b28ab-9fff-4fca-8416-ef79ecf1e78a","Type":"ContainerStarted","Data":"7634da023d0d7056ca5fc71dbc1159cbd3547588aa4167cf615c3a0077a3aab9"}
Oct 30 13:48:20.437106 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:20.437044    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq" podStartSLOduration=2.43702479 podStartE2EDuration="2.43702479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:20.435741304 +0000 UTC m=+3742.019170997" watchObservedRunningTime="2026-10-30 13:48:20.43702479 +0000 UTC m=+3742.020454481"
Oct 30 13:48:21.270318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:21.270260    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:21.275437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:21.275409    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:21.387762 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:21.387732    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:21.388799 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:21.388774    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:24.809803 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:24.809772    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"]
Oct 30 13:48:24.813676 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:24.813658    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"
Oct 30 13:48:24.827631 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:24.827609    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-f53955\""
Oct 30 13:48:24.827787 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:24.827648    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-db4e8b22-dockercfg-5b2h2\""
Oct 30 13:48:24.841170 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:24.841139    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"]
Oct 30 13:48:24.888927 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:24.888890    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"]
Oct 30 13:48:24.893950 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:24.893924    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"
Oct 30 13:48:24.917421 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:24.917394    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh\" (UID: \"e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"
Oct 30 13:48:24.917586 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:24.917462    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmb2w\" (UniqueName: \"kubernetes.io/projected/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507-kube-api-access-tmb2w\") pod \"payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh\" (UID: \"e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"
Oct 30 13:48:24.917999 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:24.917976    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"]
Oct 30 13:48:25.019200 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:25.019165    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13707505-470c-4de3-9d0d-1c4c039ae93e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5\" (UID: \"13707505-470c-4de3-9d0d-1c4c039ae93e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"
Oct 30 13:48:25.019448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:25.019316    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh\" (UID: \"e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"
Oct 30 13:48:25.019448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:25.019390    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmb2w\" (UniqueName: \"kubernetes.io/projected/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507-kube-api-access-tmb2w\") pod \"payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh\" (UID: \"e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"
Oct 30 13:48:25.019601 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:25.019509    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxzgn\" (UniqueName: \"kubernetes.io/projected/13707505-470c-4de3-9d0d-1c4c039ae93e-kube-api-access-lxzgn\") pod \"payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5\" (UID: \"13707505-470c-4de3-9d0d-1c4c039ae93e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"
Oct 30 13:48:25.020070 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:25.020049    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh\" (UID: \"e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"
Oct 30 13:48:25.042200 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:25.042172    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmb2w\" (UniqueName: \"kubernetes.io/projected/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507-kube-api-access-tmb2w\") pod \"payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh\" (UID: \"e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"
Oct 30 13:48:25.120990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:25.120882    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxzgn\" (UniqueName: \"kubernetes.io/projected/13707505-470c-4de3-9d0d-1c4c039ae93e-kube-api-access-lxzgn\") pod \"payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5\" (UID: \"13707505-470c-4de3-9d0d-1c4c039ae93e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"
Oct 30 13:48:25.120990 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:25.120946    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13707505-470c-4de3-9d0d-1c4c039ae93e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5\" (UID: \"13707505-470c-4de3-9d0d-1c4c039ae93e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"
Oct 30 13:48:25.121697 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:25.121679    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13707505-470c-4de3-9d0d-1c4c039ae93e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5\" (UID: \"13707505-470c-4de3-9d0d-1c4c039ae93e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"
Oct 30 13:48:25.127632 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:25.127614    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"
Oct 30 13:48:25.143029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:25.142998    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxzgn\" (UniqueName: \"kubernetes.io/projected/13707505-470c-4de3-9d0d-1c4c039ae93e-kube-api-access-lxzgn\") pod \"payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5\" (UID: \"13707505-470c-4de3-9d0d-1c4c039ae93e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"
Oct 30 13:48:25.210912 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:25.210656    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"
Oct 30 13:48:25.433307 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:25.433196    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"]
Oct 30 13:48:25.436924 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:48:25.436893    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c9cf2e_972c_4bc4_bdd7_1a4c6d83d507.slice/crio-7fac047808794ab0a25570a7b041a8a3118bfd26c0fbaf142332db0a9e3efce7 WatchSource:0}: Error finding container 7fac047808794ab0a25570a7b041a8a3118bfd26c0fbaf142332db0a9e3efce7: Status 404 returned error can't find the container with id 7fac047808794ab0a25570a7b041a8a3118bfd26c0fbaf142332db0a9e3efce7
Oct 30 13:48:25.524253 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:25.524219    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"]
Oct 30 13:48:25.528326 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:48:25.528293    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13707505_470c_4de3_9d0d_1c4c039ae93e.slice/crio-6f68d6f991a50f3fb6fc2b109e31b5ef4771f9b9807d43fa442510e93f8064df WatchSource:0}: Error finding container 6f68d6f991a50f3fb6fc2b109e31b5ef4771f9b9807d43fa442510e93f8064df: Status 404 returned error can't find the container with id 6f68d6f991a50f3fb6fc2b109e31b5ef4771f9b9807d43fa442510e93f8064df
Oct 30 13:48:26.412667 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:26.412607    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5" event={"ID":"13707505-470c-4de3-9d0d-1c4c039ae93e","Type":"ContainerStarted","Data":"cb7a50ddf3e257976132f6a00a76d00dac22749a25bcb8a0c6acee8b13463dde"}
Oct 30 13:48:26.413158 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:26.412723    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"
Oct 30 13:48:26.413158 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:26.412756    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5" event={"ID":"13707505-470c-4de3-9d0d-1c4c039ae93e","Type":"ContainerStarted","Data":"6f68d6f991a50f3fb6fc2b109e31b5ef4771f9b9807d43fa442510e93f8064df"}
Oct 30 13:48:26.414246 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:26.414220    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh" event={"ID":"e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507","Type":"ContainerStarted","Data":"efb6750fff6d513f1a2addea79a5c73ce6cf9d05b11b82c15f51d4c993584f3b"}
Oct 30 13:48:26.414395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:26.414299    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh" event={"ID":"e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507","Type":"ContainerStarted","Data":"7fac047808794ab0a25570a7b041a8a3118bfd26c0fbaf142332db0a9e3efce7"}
Oct 30 13:48:26.414395 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:26.414344    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"
Oct 30 13:48:26.450994 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:26.450844    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5" podStartSLOduration=2.450827792 podStartE2EDuration="2.450827792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:26.45067419 +0000 UTC m=+3748.034103909" watchObservedRunningTime="2026-10-30 13:48:26.450827792 +0000 UTC m=+3748.034257487"
Oct 30 13:48:26.487827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:26.487777    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh" podStartSLOduration=2.487757545 podStartE2EDuration="2.487757545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:26.485997711 +0000 UTC m=+3748.069427424" watchObservedRunningTime="2026-10-30 13:48:26.487757545 +0000 UTC m=+3748.071187230"
Oct 30 13:48:37.422916 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:37.422885    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"
Oct 30 13:48:37.424490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:37.422965    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"
Oct 30 13:48:37.429409 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:37.429383    5893 patch_prober.go:29] interesting pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:48:37.429526 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:37.429433    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq" podUID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:48:38.500965 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:38.500932    5893 generic.go:356] "Generic (PLEG): container finished" podID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerID="9ba11186f32613d095202e52b628d530357423fc9e9d7d430734a761bc37ec11" exitCode=0
Oct 30 13:48:38.501382 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:38.501006    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq" event={"ID":"d25b28ab-9fff-4fca-8416-ef79ecf1e78a","Type":"ContainerDied","Data":"9ba11186f32613d095202e52b628d530357423fc9e9d7d430734a761bc37ec11"}
Oct 30 13:48:38.501451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:38.501436    5893 scope.go:122] "RemoveContainer" containerID="9ba11186f32613d095202e52b628d530357423fc9e9d7d430734a761bc37ec11"
Oct 30 13:48:38.502710 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:38.502688    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:48:38.502779 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:38.502756    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:48:38.502817 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:38.502788    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:48:39.509036 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:39.509004    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq" event={"ID":"d25b28ab-9fff-4fca-8416-ef79ecf1e78a","Type":"ContainerStarted","Data":"65341ccbecb51d20ae83567623d263d962dbd2cd780409fb563f54c4ade737dd"}
Oct 30 13:48:40.270792 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:40.270755    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:40.271098 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:40.270814    5893 patch_prober.go:29] interesting pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:48:40.271259 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:40.271237    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq" podUID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 13:48:41.269933 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:41.269896    5893 patch_prober.go:29] interesting pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:48:41.270407 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:41.269961    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq" podUID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 13:48:42.272652 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:42.272605    5893 patch_prober.go:29] interesting pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:48:42.273156 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:42.272664    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq" podUID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 13:48:43.270880 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:43.270839    5893 patch_prober.go:29] interesting pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:48:43.271215 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:43.271181    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq" podUID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 13:48:44.270396 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:44.270364    5893 patch_prober.go:29] interesting pod/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Oct 30 13:48:44.270795 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:44.270417    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq" podUID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Oct 30 13:48:45.273791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:45.273765    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:45.274325 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:45.274104    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:45.274860 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:48:45.274845    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:48:48.645113 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:48:48.645073    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:49:02.145417 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:02.145389    5893 scope.go:122] "RemoveContainer" containerID="7d4eaa450282a72d0a7505c311c91838b83e18dd31a10641b0ecf7d9e3bde631"
Oct 30 13:49:18.649120 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:49:18.649071    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:49:25.541893 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:25.541861    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"]
Oct 30 13:49:25.578736 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:25.578701    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"]
Oct 30 13:49:30.542239 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:30.542191    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5" podUID="13707505-470c-4de3-9d0d-1c4c039ae93e" containerName="payload-processing" containerID="cri-o://cb7a50ddf3e257976132f6a00a76d00dac22749a25bcb8a0c6acee8b13463dde" gracePeriod=25
Oct 30 13:49:30.579216 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:30.579167    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh" podUID="e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507" containerName="payload-pre-processing" containerID="cri-o://efb6750fff6d513f1a2addea79a5c73ce6cf9d05b11b82c15f51d4c993584f3b" gracePeriod=25
Oct 30 13:49:34.510921 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:34.510888    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"]
Oct 30 13:49:34.511348 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:34.511135    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq" podUID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerName="istio-proxy" containerID="cri-o://65341ccbecb51d20ae83567623d263d962dbd2cd780409fb563f54c4ade737dd" gracePeriod=30
Oct 30 13:49:37.420487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:37.420440    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh" podUID="e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:49:37.420487 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:37.420467    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5" podUID="13707505-470c-4de3-9d0d-1c4c039ae93e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:49:39.768504 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.768478    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:49:39.832852 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.832755    5893 generic.go:356] "Generic (PLEG): container finished" podID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerID="65341ccbecb51d20ae83567623d263d962dbd2cd780409fb563f54c4ade737dd" exitCode=0
Oct 30 13:49:39.832852 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.832817    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"
Oct 30 13:49:39.833061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.832841    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq" event={"ID":"d25b28ab-9fff-4fca-8416-ef79ecf1e78a","Type":"ContainerDied","Data":"65341ccbecb51d20ae83567623d263d962dbd2cd780409fb563f54c4ade737dd"}
Oct 30 13:49:39.833061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.832915    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq" event={"ID":"d25b28ab-9fff-4fca-8416-ef79ecf1e78a","Type":"ContainerDied","Data":"7634da023d0d7056ca5fc71dbc1159cbd3547588aa4167cf615c3a0077a3aab9"}
Oct 30 13:49:39.833061 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.832947    5893 scope.go:122] "RemoveContainer" containerID="65341ccbecb51d20ae83567623d263d962dbd2cd780409fb563f54c4ade737dd"
Oct 30 13:49:39.844397 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.844373    5893 scope.go:122] "RemoveContainer" containerID="9ba11186f32613d095202e52b628d530357423fc9e9d7d430734a761bc37ec11"
Oct 30 13:49:39.855231 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.855198    5893 scope.go:122] "RemoveContainer" containerID="65341ccbecb51d20ae83567623d263d962dbd2cd780409fb563f54c4ade737dd"
Oct 30 13:49:39.855624 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:49:39.855598    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65341ccbecb51d20ae83567623d263d962dbd2cd780409fb563f54c4ade737dd\": container with ID starting with 65341ccbecb51d20ae83567623d263d962dbd2cd780409fb563f54c4ade737dd not found: ID does not exist" containerID="65341ccbecb51d20ae83567623d263d962dbd2cd780409fb563f54c4ade737dd"
Oct 30 13:49:39.855704 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.855633    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65341ccbecb51d20ae83567623d263d962dbd2cd780409fb563f54c4ade737dd"} err="failed to get container status \"65341ccbecb51d20ae83567623d263d962dbd2cd780409fb563f54c4ade737dd\": rpc error: code = NotFound desc = could not find container \"65341ccbecb51d20ae83567623d263d962dbd2cd780409fb563f54c4ade737dd\": container with ID starting with 65341ccbecb51d20ae83567623d263d962dbd2cd780409fb563f54c4ade737dd not found: ID does not exist"
Oct 30 13:49:39.855704 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.855653    5893 scope.go:122] "RemoveContainer" containerID="9ba11186f32613d095202e52b628d530357423fc9e9d7d430734a761bc37ec11"
Oct 30 13:49:39.856010 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:49:39.855993    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ba11186f32613d095202e52b628d530357423fc9e9d7d430734a761bc37ec11\": container with ID starting with 9ba11186f32613d095202e52b628d530357423fc9e9d7d430734a761bc37ec11 not found: ID does not exist" containerID="9ba11186f32613d095202e52b628d530357423fc9e9d7d430734a761bc37ec11"
Oct 30 13:49:39.856072 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.856013    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ba11186f32613d095202e52b628d530357423fc9e9d7d430734a761bc37ec11"} err="failed to get container status \"9ba11186f32613d095202e52b628d530357423fc9e9d7d430734a761bc37ec11\": rpc error: code = NotFound desc = could not find container \"9ba11186f32613d095202e52b628d530357423fc9e9d7d430734a761bc37ec11\": container with ID starting with 9ba11186f32613d095202e52b628d530357423fc9e9d7d430734a761bc37ec11 not found: ID does not exist"
Oct 30 13:49:39.917490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.917454    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-data\") pod \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") "
Oct 30 13:49:39.917700 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.917510    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-token\" (UniqueName: \"kubernetes.io/projected/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-token\") pod \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") "
Oct 30 13:49:39.917700 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.917558    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-envoy\") pod \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") "
Oct 30 13:49:39.917700 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.917638    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-credential-socket\") pod \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") "
Oct 30 13:49:39.917700 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.917663    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-workload-certs\") pod \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") "
Oct 30 13:49:39.917700 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.917690    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-workload-socket\") pod \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") "
Oct 30 13:49:39.917987 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.917737    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istiod-ca-cert\") pod \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") "
Oct 30 13:49:39.917987 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.917814    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-data" pod "d25b28ab-9fff-4fca-8416-ef79ecf1e78a" (UID: "d25b28ab-9fff-4fca-8416-ef79ecf1e78a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:49:39.917987 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.917838    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-kube-api-access-xf8kt\" (UniqueName: \"kubernetes.io/projected/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-kube-api-access-xf8kt\") pod \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") "
Oct 30 13:49:39.917987 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.917906    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-podinfo\") pod \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\" (UID: \"d25b28ab-9fff-4fca-8416-ef79ecf1e78a\") "
Oct 30 13:49:39.918223 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.918010    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-credential-socket" pod "d25b28ab-9fff-4fca-8416-ef79ecf1e78a" (UID: "d25b28ab-9fff-4fca-8416-ef79ecf1e78a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:49:39.918223 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.918049    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-workload-socket" pod "d25b28ab-9fff-4fca-8416-ef79ecf1e78a" (UID: "d25b28ab-9fff-4fca-8416-ef79ecf1e78a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:49:39.918383 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.918220    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istiod-ca-cert" pod "d25b28ab-9fff-4fca-8416-ef79ecf1e78a" (UID: "d25b28ab-9fff-4fca-8416-ef79ecf1e78a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:39.918383 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.918334    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-workload-certs" pod "d25b28ab-9fff-4fca-8416-ef79ecf1e78a" (UID: "d25b28ab-9fff-4fca-8416-ef79ecf1e78a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:49:39.918501 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.918427    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:39.918501 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.918447    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:39.918501 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.918463    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:39.918501 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.918478    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:39.920084 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.920065    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-token" pod "d25b28ab-9fff-4fca-8416-ef79ecf1e78a" (UID: "d25b28ab-9fff-4fca-8416-ef79ecf1e78a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:39.920232 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.920210    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-podinfo" pod "d25b28ab-9fff-4fca-8416-ef79ecf1e78a" (UID: "d25b28ab-9fff-4fca-8416-ef79ecf1e78a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:49:39.920571 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.920545    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-envoy" pod "d25b28ab-9fff-4fca-8416-ef79ecf1e78a" (UID: "d25b28ab-9fff-4fca-8416-ef79ecf1e78a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:49:39.920622 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:39.920590    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-kube-api-access-xf8kt" pod "d25b28ab-9fff-4fca-8416-ef79ecf1e78a" (UID: "d25b28ab-9fff-4fca-8416-ef79ecf1e78a"). InnerVolumeSpecName "kube-api-access-xf8kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:40.019882 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:40.019837    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:40.019882 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:40.019872    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:40.019882 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:40.019894    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:40.020150 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:40.019905    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:40.020150 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:40.019917    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf8kt\" (UniqueName: \"kubernetes.io/projected/d25b28ab-9fff-4fca-8416-ef79ecf1e78a-kube-api-access-xf8kt\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:40.174868 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:40.174828    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"]
Oct 30 13:49:40.184825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:40.184784    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f53955-openshift-default-6c7b767cc79fbmq"]
Oct 30 13:49:40.530377 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:40.530346    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" path="/var/lib/kubelet/pods/d25b28ab-9fff-4fca-8416-ef79ecf1e78a/volumes"
Oct 30 13:49:47.420769 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:47.420725    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh" podUID="e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:49:47.421203 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:47.420725    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5" podUID="13707505-470c-4de3-9d0d-1c4c039ae93e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:49:48.651183 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:49:48.651147    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:49:50.833580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.833555    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"
Oct 30 13:49:50.859063 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.859040    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"
Oct 30 13:49:50.891069 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.891031    5893 generic.go:356] "Generic (PLEG): container finished" podID="13707505-470c-4de3-9d0d-1c4c039ae93e" containerID="cb7a50ddf3e257976132f6a00a76d00dac22749a25bcb8a0c6acee8b13463dde" exitCode=0
Oct 30 13:49:50.891261 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.891075    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"
Oct 30 13:49:50.891261 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.891116    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5" event={"ID":"13707505-470c-4de3-9d0d-1c4c039ae93e","Type":"ContainerDied","Data":"cb7a50ddf3e257976132f6a00a76d00dac22749a25bcb8a0c6acee8b13463dde"}
Oct 30 13:49:50.891261 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.891195    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5" event={"ID":"13707505-470c-4de3-9d0d-1c4c039ae93e","Type":"ContainerDied","Data":"6f68d6f991a50f3fb6fc2b109e31b5ef4771f9b9807d43fa442510e93f8064df"}
Oct 30 13:49:50.891261 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.891231    5893 scope.go:122] "RemoveContainer" containerID="cb7a50ddf3e257976132f6a00a76d00dac22749a25bcb8a0c6acee8b13463dde"
Oct 30 13:49:50.892822 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.892800    5893 generic.go:356] "Generic (PLEG): container finished" podID="e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507" containerID="efb6750fff6d513f1a2addea79a5c73ce6cf9d05b11b82c15f51d4c993584f3b" exitCode=0
Oct 30 13:49:50.892929 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.892853    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"
Oct 30 13:49:50.893069 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.893046    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh" event={"ID":"e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507","Type":"ContainerDied","Data":"efb6750fff6d513f1a2addea79a5c73ce6cf9d05b11b82c15f51d4c993584f3b"}
Oct 30 13:49:50.893112 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.893101    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh" event={"ID":"e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507","Type":"ContainerDied","Data":"7fac047808794ab0a25570a7b041a8a3118bfd26c0fbaf142332db0a9e3efce7"}
Oct 30 13:49:50.900064 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.900047    5893 scope.go:122] "RemoveContainer" containerID="cb7a50ddf3e257976132f6a00a76d00dac22749a25bcb8a0c6acee8b13463dde"
Oct 30 13:49:50.900340 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:49:50.900320    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7a50ddf3e257976132f6a00a76d00dac22749a25bcb8a0c6acee8b13463dde\": container with ID starting with cb7a50ddf3e257976132f6a00a76d00dac22749a25bcb8a0c6acee8b13463dde not found: ID does not exist" containerID="cb7a50ddf3e257976132f6a00a76d00dac22749a25bcb8a0c6acee8b13463dde"
Oct 30 13:49:50.900411 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.900348    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7a50ddf3e257976132f6a00a76d00dac22749a25bcb8a0c6acee8b13463dde"} err="failed to get container status \"cb7a50ddf3e257976132f6a00a76d00dac22749a25bcb8a0c6acee8b13463dde\": rpc error: code = NotFound desc = could not find container \"cb7a50ddf3e257976132f6a00a76d00dac22749a25bcb8a0c6acee8b13463dde\": container with ID starting with cb7a50ddf3e257976132f6a00a76d00dac22749a25bcb8a0c6acee8b13463dde not found: ID does not exist"
Oct 30 13:49:50.900411 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.900367    5893 scope.go:122] "RemoveContainer" containerID="efb6750fff6d513f1a2addea79a5c73ce6cf9d05b11b82c15f51d4c993584f3b"
Oct 30 13:49:50.910386 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.910039    5893 scope.go:122] "RemoveContainer" containerID="efb6750fff6d513f1a2addea79a5c73ce6cf9d05b11b82c15f51d4c993584f3b"
Oct 30 13:49:50.910566 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:49:50.910543    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb6750fff6d513f1a2addea79a5c73ce6cf9d05b11b82c15f51d4c993584f3b\": container with ID starting with efb6750fff6d513f1a2addea79a5c73ce6cf9d05b11b82c15f51d4c993584f3b not found: ID does not exist" containerID="efb6750fff6d513f1a2addea79a5c73ce6cf9d05b11b82c15f51d4c993584f3b"
Oct 30 13:49:50.910646 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.910574    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb6750fff6d513f1a2addea79a5c73ce6cf9d05b11b82c15f51d4c993584f3b"} err="failed to get container status \"efb6750fff6d513f1a2addea79a5c73ce6cf9d05b11b82c15f51d4c993584f3b\": rpc error: code = NotFound desc = could not find container \"efb6750fff6d513f1a2addea79a5c73ce6cf9d05b11b82c15f51d4c993584f3b\": container with ID starting with efb6750fff6d513f1a2addea79a5c73ce6cf9d05b11b82c15f51d4c993584f3b not found: ID does not exist"
Oct 30 13:49:50.932007 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.931967    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13707505-470c-4de3-9d0d-1c4c039ae93e-kube-api-access-lxzgn\" (UniqueName: \"kubernetes.io/projected/13707505-470c-4de3-9d0d-1c4c039ae93e-kube-api-access-lxzgn\") pod \"13707505-470c-4de3-9d0d-1c4c039ae93e\" (UID: \"13707505-470c-4de3-9d0d-1c4c039ae93e\") "
Oct 30 13:49:50.932165 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.932016    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507-plugins-config-volume\") pod \"e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507\" (UID: \"e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507\") "
Oct 30 13:49:50.932165 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.932099    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507-kube-api-access-tmb2w\" (UniqueName: \"kubernetes.io/projected/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507-kube-api-access-tmb2w\") pod \"e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507\" (UID: \"e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507\") "
Oct 30 13:49:50.932165 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.932141    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13707505-470c-4de3-9d0d-1c4c039ae93e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13707505-470c-4de3-9d0d-1c4c039ae93e-plugins-config-volume\") pod \"13707505-470c-4de3-9d0d-1c4c039ae93e\" (UID: \"13707505-470c-4de3-9d0d-1c4c039ae93e\") "
Oct 30 13:49:50.932506 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.932476    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507-plugins-config-volume" pod "e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507" (UID: "e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:50.932634 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.932604    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13707505-470c-4de3-9d0d-1c4c039ae93e-plugins-config-volume" pod "13707505-470c-4de3-9d0d-1c4c039ae93e" (UID: "13707505-470c-4de3-9d0d-1c4c039ae93e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:50.934460 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.934441    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507-kube-api-access-tmb2w" pod "e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507" (UID: "e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507"). InnerVolumeSpecName "kube-api-access-tmb2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:50.934811 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:50.934788    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13707505-470c-4de3-9d0d-1c4c039ae93e-kube-api-access-lxzgn" pod "13707505-470c-4de3-9d0d-1c4c039ae93e" (UID: "13707505-470c-4de3-9d0d-1c4c039ae93e"). InnerVolumeSpecName "kube-api-access-lxzgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:51.032986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:51.032954    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmb2w\" (UniqueName: \"kubernetes.io/projected/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507-kube-api-access-tmb2w\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:51.032986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:51.032983    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13707505-470c-4de3-9d0d-1c4c039ae93e-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:51.032986 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:51.032992    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxzgn\" (UniqueName: \"kubernetes.io/projected/13707505-470c-4de3-9d0d-1c4c039ae93e-kube-api-access-lxzgn\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:51.033258 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:51.033003    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:51.237527 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:51.237493    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"]
Oct 30 13:49:51.247368 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:51.247342    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f53955-9c495qkrmh"]
Oct 30 13:49:51.270966 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:51.270934    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"]
Oct 30 13:49:51.274604 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:51.274581    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f53955-7bd7df6542pbc5"]
Oct 30 13:49:52.530563 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:52.530532    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13707505-470c-4de3-9d0d-1c4c039ae93e" path="/var/lib/kubelet/pods/13707505-470c-4de3-9d0d-1c4c039ae93e/volumes"
Oct 30 13:49:52.530964 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:52.530906    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507" path="/var/lib/kubelet/pods/e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507/volumes"
Oct 30 13:49:54.091120 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.091083    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"]
Oct 30 13:49:54.091805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.091747    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh" podUID="520c958e-79f7-4563-af5d-3f313563da44" containerName="manager" containerID="cri-o://621b19b9366cb91baa2bdb300ab8efefb67be776578e4449e8bd4a9f92083850" gracePeriod=10
Oct 30 13:49:54.360752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.360723    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"
Oct 30 13:49:54.469648 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.469617    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/520c958e-79f7-4563-af5d-3f313563da44-kube-api-access-hk46x\" (UniqueName: \"kubernetes.io/projected/520c958e-79f7-4563-af5d-3f313563da44-kube-api-access-hk46x\") pod \"520c958e-79f7-4563-af5d-3f313563da44\" (UID: \"520c958e-79f7-4563-af5d-3f313563da44\") "
Oct 30 13:49:54.469849 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.469703    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/520c958e-79f7-4563-af5d-3f313563da44-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/520c958e-79f7-4563-af5d-3f313563da44-extensions-socket-volume\") pod \"520c958e-79f7-4563-af5d-3f313563da44\" (UID: \"520c958e-79f7-4563-af5d-3f313563da44\") "
Oct 30 13:49:54.470058 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.470033    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/520c958e-79f7-4563-af5d-3f313563da44-extensions-socket-volume" pod "520c958e-79f7-4563-af5d-3f313563da44" (UID: "520c958e-79f7-4563-af5d-3f313563da44"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:49:54.472046 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.472023    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/520c958e-79f7-4563-af5d-3f313563da44-kube-api-access-hk46x" pod "520c958e-79f7-4563-af5d-3f313563da44" (UID: "520c958e-79f7-4563-af5d-3f313563da44"). InnerVolumeSpecName "kube-api-access-hk46x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:54.570734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.570695    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk46x\" (UniqueName: \"kubernetes.io/projected/520c958e-79f7-4563-af5d-3f313563da44-kube-api-access-hk46x\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:54.570734 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.570733    5893 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/520c958e-79f7-4563-af5d-3f313563da44-extensions-socket-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:49:54.915662 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.915629    5893 generic.go:356] "Generic (PLEG): container finished" podID="520c958e-79f7-4563-af5d-3f313563da44" containerID="621b19b9366cb91baa2bdb300ab8efefb67be776578e4449e8bd4a9f92083850" exitCode=0
Oct 30 13:49:54.915864 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.915678    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"
Oct 30 13:49:54.915864 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.915689    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh" event={"ID":"520c958e-79f7-4563-af5d-3f313563da44","Type":"ContainerDied","Data":"621b19b9366cb91baa2bdb300ab8efefb67be776578e4449e8bd4a9f92083850"}
Oct 30 13:49:54.915864 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.915741    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh" event={"ID":"520c958e-79f7-4563-af5d-3f313563da44","Type":"ContainerDied","Data":"2f9546283bb14118565cf439a0a4fe0c6987c4729635ea776424d2058a6f4346"}
Oct 30 13:49:54.915864 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.915764    5893 scope.go:122] "RemoveContainer" containerID="621b19b9366cb91baa2bdb300ab8efefb67be776578e4449e8bd4a9f92083850"
Oct 30 13:49:54.925611 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.925589    5893 scope.go:122] "RemoveContainer" containerID="621b19b9366cb91baa2bdb300ab8efefb67be776578e4449e8bd4a9f92083850"
Oct 30 13:49:54.925861 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:49:54.925842    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621b19b9366cb91baa2bdb300ab8efefb67be776578e4449e8bd4a9f92083850\": container with ID starting with 621b19b9366cb91baa2bdb300ab8efefb67be776578e4449e8bd4a9f92083850 not found: ID does not exist" containerID="621b19b9366cb91baa2bdb300ab8efefb67be776578e4449e8bd4a9f92083850"
Oct 30 13:49:54.925911 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.925870    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621b19b9366cb91baa2bdb300ab8efefb67be776578e4449e8bd4a9f92083850"} err="failed to get container status \"621b19b9366cb91baa2bdb300ab8efefb67be776578e4449e8bd4a9f92083850\": rpc error: code = NotFound desc = could not find container \"621b19b9366cb91baa2bdb300ab8efefb67be776578e4449e8bd4a9f92083850\": container with ID starting with 621b19b9366cb91baa2bdb300ab8efefb67be776578e4449e8bd4a9f92083850 not found: ID does not exist"
Oct 30 13:49:54.961794 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.961759    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"]
Oct 30 13:49:54.971827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:54.971804    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-zq9bh"]
Oct 30 13:49:56.529859 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:49:56.529829    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="520c958e-79f7-4563-af5d-3f313563da44" path="/var/lib/kubelet/pods/520c958e-79f7-4563-af5d-3f313563da44/volumes"
Oct 30 13:50:18.644624 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:50:18.644588    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:50:48.645306 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:50:48.645203    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:50:55.275159 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.275112    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn"]
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276132    5893 cpu_manager.go:414] "containerMap: removing container" podUID="e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507" containerName="payload-pre-processing"
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276158    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507" containerName="payload-pre-processing"
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276194    5893 cpu_manager.go:414] "containerMap: removing container" podUID="520c958e-79f7-4563-af5d-3f313563da44" containerName="manager"
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276201    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="520c958e-79f7-4563-af5d-3f313563da44" containerName="manager"
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276260    5893 cpu_manager.go:414] "containerMap: removing container" podUID="13707505-470c-4de3-9d0d-1c4c039ae93e" containerName="payload-processing"
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276298    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13707505-470c-4de3-9d0d-1c4c039ae93e" containerName="payload-processing"
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276320    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerName="istio-proxy"
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276326    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerName="istio-proxy"
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276350    5893 cpu_manager.go:414] "containerMap: removing container" podUID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerName="istio-proxy"
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276356    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerName="istio-proxy"
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276545    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="13707505-470c-4de3-9d0d-1c4c039ae93e" containerName="payload-processing"
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276557    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerName="istio-proxy"
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276574    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25b28ab-9fff-4fca-8416-ef79ecf1e78a" containerName="istio-proxy"
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276588    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5c9cf2e-972c-4bc4-bdd7-1a4c6d83d507" containerName="payload-pre-processing"
Oct 30 13:50:55.279917 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.276606    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="520c958e-79f7-4563-af5d-3f313563da44" containerName="manager"
Oct 30 13:50:55.286092 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.286055    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn"
Oct 30 13:50:55.300288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.300247    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn"]
Oct 30 13:50:55.307609 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.307575    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-k8kf6\""
Oct 30 13:50:55.386905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.386853    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/446440ca-255b-4262-8c8f-76953e77f832-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-kc2vn\" (UID: \"446440ca-255b-4262-8c8f-76953e77f832\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn"
Oct 30 13:50:55.387099 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.386911    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9t7k\" (UniqueName: \"kubernetes.io/projected/446440ca-255b-4262-8c8f-76953e77f832-kube-api-access-p9t7k\") pod \"kuadrant-operator-controller-manager-57bb9bfff-kc2vn\" (UID: \"446440ca-255b-4262-8c8f-76953e77f832\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn"
Oct 30 13:50:55.487668 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.487635    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/446440ca-255b-4262-8c8f-76953e77f832-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-kc2vn\" (UID: \"446440ca-255b-4262-8c8f-76953e77f832\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn"
Oct 30 13:50:55.487668 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.487673    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9t7k\" (UniqueName: \"kubernetes.io/projected/446440ca-255b-4262-8c8f-76953e77f832-kube-api-access-p9t7k\") pod \"kuadrant-operator-controller-manager-57bb9bfff-kc2vn\" (UID: \"446440ca-255b-4262-8c8f-76953e77f832\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn"
Oct 30 13:50:55.488039 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.488018    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/446440ca-255b-4262-8c8f-76953e77f832-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-57bb9bfff-kc2vn\" (UID: \"446440ca-255b-4262-8c8f-76953e77f832\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn"
Oct 30 13:50:55.511814 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.511778    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9t7k\" (UniqueName: \"kubernetes.io/projected/446440ca-255b-4262-8c8f-76953e77f832-kube-api-access-p9t7k\") pod \"kuadrant-operator-controller-manager-57bb9bfff-kc2vn\" (UID: \"446440ca-255b-4262-8c8f-76953e77f832\") " pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn"
Oct 30 13:50:55.599618 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.599546    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn"
Oct 30 13:50:55.928584 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:55.928500    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn"]
Oct 30 13:50:55.933983 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:50:55.933959    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod446440ca_255b_4262_8c8f_76953e77f832.slice/crio-e69e36ff0a2bf748c5cedbf55daf04b335a4022fceb5f350ee2b42b92464223d WatchSource:0}: Error finding container e69e36ff0a2bf748c5cedbf55daf04b335a4022fceb5f350ee2b42b92464223d: Status 404 returned error can't find the container with id e69e36ff0a2bf748c5cedbf55daf04b335a4022fceb5f350ee2b42b92464223d
Oct 30 13:50:56.199212 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:56.199136    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn" event={"ID":"446440ca-255b-4262-8c8f-76953e77f832","Type":"ContainerStarted","Data":"de6db00ae48ee4abe032e58f760f5229c51b3031aa748c48b8404d34eb829ed7"}
Oct 30 13:50:56.199212 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:56.199187    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn" event={"ID":"446440ca-255b-4262-8c8f-76953e77f832","Type":"ContainerStarted","Data":"e69e36ff0a2bf748c5cedbf55daf04b335a4022fceb5f350ee2b42b92464223d"}
Oct 30 13:50:56.199438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:56.199253    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn"
Oct 30 13:50:56.246722 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:50:56.246659    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn" podStartSLOduration=1.2466431789999999 podStartE2EDuration="1.246643179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:56.245575389 +0000 UTC m=+3897.829005084" watchObservedRunningTime="2026-10-30 13:50:56.246643179 +0000 UTC m=+3897.830072876"
Oct 30 13:51:01.470605 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:01.470575    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:51:01.482900 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:01.482874    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:51:01.490819 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:01.490796    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:51:01.505963 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:01.505936    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:51:07.206292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:07.206242    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-57bb9bfff-kc2vn"
Oct 30 13:51:18.645104 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:51:18.645056    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:51:47.069044 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.068999    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"]
Oct 30 13:51:47.072642 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.072626    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.089972 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.089951    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-bf43af-openshift-default-dockercfg-n82cw\""
Oct 30 13:51:47.107594 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.107560    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"]
Oct 30 13:51:47.166458 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.166414    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-credential-socket\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.166654 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.166545    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-token\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.166654 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.166582    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-data\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.166778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.166676    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istiod-ca-cert\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.166778 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.166713    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-workload-certs\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.166858 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.166798    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-envoy\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.166904 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.166860    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s95v\" (UniqueName: \"kubernetes.io/projected/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-kube-api-access-8s95v\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.166904 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.166884    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-workload-socket\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.167008 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.166986    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-podinfo\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.268524 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.268492    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-token\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.268524 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.268526    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-data\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.268801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.268563    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istiod-ca-cert\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.268801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.268590    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-workload-certs\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.268801 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.268630    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-envoy\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.269002 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.268812    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s95v\" (UniqueName: \"kubernetes.io/projected/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-kube-api-access-8s95v\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.269002 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.268871    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-workload-socket\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.269002 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.268912    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-podinfo\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.269002 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.268977    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-credential-socket\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.269250 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.269003    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-data\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.269250 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.269031    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-workload-certs\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.269389 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.269323    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-credential-socket\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.269464 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.269435    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istiod-ca-cert\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.269464 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.269441    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-workload-socket\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.271181 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.271153    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-envoy\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.271518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.271500    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-podinfo\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.290035 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.290007    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-token\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.290202 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.290090    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s95v\" (UniqueName: \"kubernetes.io/projected/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-kube-api-access-8s95v\") pod \"e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") " pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.386111 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.386021    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:47.713127 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.713023    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"]
Oct 30 13:51:47.719437 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:51:47.719402    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84f1ca9_009d_4c67_af4c_6cbd9c13ef0a.slice/crio-fbbe6ae455dc97bb85a3554f8accd0ebdb10163b3281a3824e9b725e671d4ca0 WatchSource:0}: Error finding container fbbe6ae455dc97bb85a3554f8accd0ebdb10163b3281a3824e9b725e671d4ca0: Status 404 returned error can't find the container with id fbbe6ae455dc97bb85a3554f8accd0ebdb10163b3281a3824e9b725e671d4ca0
Oct 30 13:51:47.721832 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.721796    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:51:47.721925 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.721871    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:51:47.721925 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:47.721897    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:51:48.506417 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:48.506378    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6" event={"ID":"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a","Type":"ContainerStarted","Data":"46718dd9451cc266fbd615050b0a8c904b7fb714fe646c285f5ea89f61b1d5c4"}
Oct 30 13:51:48.506825 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:48.506438    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6" event={"ID":"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a","Type":"ContainerStarted","Data":"fbbe6ae455dc97bb85a3554f8accd0ebdb10163b3281a3824e9b725e671d4ca0"}
Oct 30 13:51:48.556735 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:48.556675    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6" podStartSLOduration=1.556656958 podStartE2EDuration="1.556656958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:48.556142832 +0000 UTC m=+3950.139572555" watchObservedRunningTime="2026-10-30 13:51:48.556656958 +0000 UTC m=+3950.140086654"
Oct 30 13:51:48.644946 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:51:48.644908    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:51:49.387308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:49.387246    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:49.393064 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:49.393032    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:49.511987 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:49.511954    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:49.513010 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:49.512989    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:51:53.048713 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.048681    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"]
Oct 30 13:51:53.052486 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.052467    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"
Oct 30 13:51:53.074190 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.074170    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-b298e363-dockercfg-khsw6\""
Oct 30 13:51:53.084476 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.084449    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-bf43af\""
Oct 30 13:51:53.085980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.085956    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"]
Oct 30 13:51:53.131412 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.131377    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5b52\" (UniqueName: \"kubernetes.io/projected/ca4c5a90-e1f3-4723-a12b-ca85b8508221-kube-api-access-k5b52\") pod \"payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k\" (UID: \"ca4c5a90-e1f3-4723-a12b-ca85b8508221\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"
Oct 30 13:51:53.131588 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.131424    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4c5a90-e1f3-4723-a12b-ca85b8508221-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k\" (UID: \"ca4c5a90-e1f3-4723-a12b-ca85b8508221\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"
Oct 30 13:51:53.175122 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.175057    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"]
Oct 30 13:51:53.179610 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.179586    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"
Oct 30 13:51:53.201980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.201944    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"]
Oct 30 13:51:53.232308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.232260    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5b52\" (UniqueName: \"kubernetes.io/projected/ca4c5a90-e1f3-4723-a12b-ca85b8508221-kube-api-access-k5b52\") pod \"payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k\" (UID: \"ca4c5a90-e1f3-4723-a12b-ca85b8508221\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"
Oct 30 13:51:53.232492 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.232319    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4c5a90-e1f3-4723-a12b-ca85b8508221-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k\" (UID: \"ca4c5a90-e1f3-4723-a12b-ca85b8508221\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"
Oct 30 13:51:53.233035 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.233004    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4c5a90-e1f3-4723-a12b-ca85b8508221-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k\" (UID: \"ca4c5a90-e1f3-4723-a12b-ca85b8508221\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"
Oct 30 13:51:53.253464 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.253438    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5b52\" (UniqueName: \"kubernetes.io/projected/ca4c5a90-e1f3-4723-a12b-ca85b8508221-kube-api-access-k5b52\") pod \"payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k\" (UID: \"ca4c5a90-e1f3-4723-a12b-ca85b8508221\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"
Oct 30 13:51:53.333897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.333810    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ef5379-55bf-4488-9caf-f2bcf04cf941-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw\" (UID: \"18ef5379-55bf-4488-9caf-f2bcf04cf941\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"
Oct 30 13:51:53.333897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.333869    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqcxx\" (UniqueName: \"kubernetes.io/projected/18ef5379-55bf-4488-9caf-f2bcf04cf941-kube-api-access-lqcxx\") pod \"payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw\" (UID: \"18ef5379-55bf-4488-9caf-f2bcf04cf941\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"
Oct 30 13:51:53.365640 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.365602    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"
Oct 30 13:51:53.436315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.435072    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqcxx\" (UniqueName: \"kubernetes.io/projected/18ef5379-55bf-4488-9caf-f2bcf04cf941-kube-api-access-lqcxx\") pod \"payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw\" (UID: \"18ef5379-55bf-4488-9caf-f2bcf04cf941\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"
Oct 30 13:51:53.436315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.435293    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ef5379-55bf-4488-9caf-f2bcf04cf941-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw\" (UID: \"18ef5379-55bf-4488-9caf-f2bcf04cf941\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"
Oct 30 13:51:53.436315 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.435972    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ef5379-55bf-4488-9caf-f2bcf04cf941-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw\" (UID: \"18ef5379-55bf-4488-9caf-f2bcf04cf941\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"
Oct 30 13:51:53.456525 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.456500    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqcxx\" (UniqueName: \"kubernetes.io/projected/18ef5379-55bf-4488-9caf-f2bcf04cf941-kube-api-access-lqcxx\") pod \"payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw\" (UID: \"18ef5379-55bf-4488-9caf-f2bcf04cf941\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"
Oct 30 13:51:53.492793 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.492766    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"
Oct 30 13:51:53.673113 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.672609    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"]
Oct 30 13:51:53.677968 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:51:53.677941    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4c5a90_e1f3_4723_a12b_ca85b8508221.slice/crio-a9aa5a85d906089dcb992571fed169e1243f716da0017862b4e5dc6875d39627 WatchSource:0}: Error finding container a9aa5a85d906089dcb992571fed169e1243f716da0017862b4e5dc6875d39627: Status 404 returned error can't find the container with id a9aa5a85d906089dcb992571fed169e1243f716da0017862b4e5dc6875d39627
Oct 30 13:51:53.806414 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:53.806380    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"]
Oct 30 13:51:53.813105 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:51:53.813065    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18ef5379_55bf_4488_9caf_f2bcf04cf941.slice/crio-c14477facd90389a60da1e3574598949091c9ace11857c64dbd6978fb6d01ea6 WatchSource:0}: Error finding container c14477facd90389a60da1e3574598949091c9ace11857c64dbd6978fb6d01ea6: Status 404 returned error can't find the container with id c14477facd90389a60da1e3574598949091c9ace11857c64dbd6978fb6d01ea6
Oct 30 13:51:54.544982 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:54.544943    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw" event={"ID":"18ef5379-55bf-4488-9caf-f2bcf04cf941","Type":"ContainerStarted","Data":"39a3d300402bee60f437297e0a665838d25deaa141714188c4036704cf60b38c"}
Oct 30 13:51:54.545540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:54.545014    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw" event={"ID":"18ef5379-55bf-4488-9caf-f2bcf04cf941","Type":"ContainerStarted","Data":"c14477facd90389a60da1e3574598949091c9ace11857c64dbd6978fb6d01ea6"}
Oct 30 13:51:54.545540 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:54.545143    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"
Oct 30 13:51:54.546895 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:54.546867    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k" event={"ID":"ca4c5a90-e1f3-4723-a12b-ca85b8508221","Type":"ContainerStarted","Data":"48af575fa7a02e41734d06979bd42325957cdd375b3d63937a306bef3f62baeb"}
Oct 30 13:51:54.547083 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:54.546917    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k" event={"ID":"ca4c5a90-e1f3-4723-a12b-ca85b8508221","Type":"ContainerStarted","Data":"a9aa5a85d906089dcb992571fed169e1243f716da0017862b4e5dc6875d39627"}
Oct 30 13:51:54.547083 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:54.546947    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"
Oct 30 13:51:54.582928 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:54.582870    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw" podStartSLOduration=1.5828524210000001 podStartE2EDuration="1.582852421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:54.582116613 +0000 UTC m=+3956.165546305" watchObservedRunningTime="2026-10-30 13:51:54.582852421 +0000 UTC m=+3956.166282116"
Oct 30 13:51:54.620851 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:51:54.620788    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k" podStartSLOduration=1.620765767 podStartE2EDuration="1.620765767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:54.618425925 +0000 UTC m=+3956.201855638" watchObservedRunningTime="2026-10-30 13:51:54.620765767 +0000 UTC m=+3956.204195458"
Oct 30 13:52:05.554446 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:05.554406    5893 patch_prober.go:29] interesting pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:52:05.554839 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:05.554461    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6" podUID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:52:05.558779 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:05.558749    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"
Oct 30 13:52:05.558951 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:05.558913    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"
Oct 30 13:52:06.625980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:06.625943    5893 generic.go:356] "Generic (PLEG): container finished" podID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerID="46718dd9451cc266fbd615050b0a8c904b7fb714fe646c285f5ea89f61b1d5c4" exitCode=0
Oct 30 13:52:06.625980 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:06.625984    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6" event={"ID":"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a","Type":"ContainerDied","Data":"46718dd9451cc266fbd615050b0a8c904b7fb714fe646c285f5ea89f61b1d5c4"}
Oct 30 13:52:06.626623 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:06.626530    5893 scope.go:122] "RemoveContainer" containerID="46718dd9451cc266fbd615050b0a8c904b7fb714fe646c285f5ea89f61b1d5c4"
Oct 30 13:52:06.627849 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:06.627817    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:52:06.627959 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:06.627891    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:52:06.627959 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:06.627924    5893 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 13:52:07.632593 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:07.632553    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6" event={"ID":"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a","Type":"ContainerStarted","Data":"d986a4564eb26a4d07eeed7c5f571e304bdb9e13ea91aca6d16ba8aa88a810d3"}
Oct 30 13:52:08.387024 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:08.386992    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:52:08.387353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:08.387331    5893 patch_prober.go:29] interesting pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 30 13:52:08.387450 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:08.387385    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6" podUID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 30 13:52:09.386758 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:09.386727    5893 patch_prober.go:29] interesting pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 30 13:52:09.387154 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:09.386776    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6" podUID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 30 13:52:10.386799 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:10.386762    5893 patch_prober.go:29] interesting pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 30 13:52:10.387222 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:10.386817    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6" podUID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 30 13:52:11.390296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:11.389652    5893 patch_prober.go:29] interesting pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 30 13:52:11.390296 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:11.389716    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6" podUID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 30 13:52:12.386485 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:12.386448    5893 patch_prober.go:29] interesting pod/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Oct 30 13:52:12.386684 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:12.386516    5893 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6" podUID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Oct 30 13:52:13.390130 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:13.390100    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:52:13.390553 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:13.390395    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:52:13.391050 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:13.391035    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:52:18.646029 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:52:18.645990    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:52:48.644503 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:52:48.644463    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:52:49.705410 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:49.705368    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"]
Oct 30 13:52:49.738129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:49.738097    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"]
Oct 30 13:52:54.706292 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:54.706220    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw" podUID="18ef5379-55bf-4488-9caf-f2bcf04cf941" containerName="payload-processing" containerID="cri-o://39a3d300402bee60f437297e0a665838d25deaa141714188c4036704cf60b38c" gracePeriod=25
Oct 30 13:52:54.739002 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:54.738949    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k" podUID="ca4c5a90-e1f3-4723-a12b-ca85b8508221" containerName="payload-pre-processing" containerID="cri-o://48af575fa7a02e41734d06979bd42325957cdd375b3d63937a306bef3f62baeb" gracePeriod=25
Oct 30 13:52:55.557516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:55.557461    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw" podUID="18ef5379-55bf-4488-9caf-f2bcf04cf941" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:52:55.558866 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:55.558833    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k" podUID="ca4c5a90-e1f3-4723-a12b-ca85b8508221" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:52:59.369108 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:59.369070    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"]
Oct 30 13:52:59.369565 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:52:59.369416    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6" podUID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerName="istio-proxy" containerID="cri-o://d986a4564eb26a4d07eeed7c5f571e304bdb9e13ea91aca6d16ba8aa88a810d3" gracePeriod=30
Oct 30 13:53:04.634977 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.634952    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:53:04.698435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.698348    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-workload-certs\") pod \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") "
Oct 30 13:53:04.698435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.698410    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-workload-socket\") pod \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") "
Oct 30 13:53:04.698716 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.698455    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-podinfo\") pod \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") "
Oct 30 13:53:04.698716 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.698477    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-token\" (UniqueName: \"kubernetes.io/projected/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-token\") pod \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") "
Oct 30 13:53:04.698716 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.698509    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-envoy\") pod \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") "
Oct 30 13:53:04.698716 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.698543    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-kube-api-access-8s95v\" (UniqueName: \"kubernetes.io/projected/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-kube-api-access-8s95v\") pod \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") "
Oct 30 13:53:04.698716 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.698613    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-credential-socket\") pod \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") "
Oct 30 13:53:04.698716 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.698667    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istiod-ca-cert\") pod \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") "
Oct 30 13:53:04.698716 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.698669    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-workload-certs" pod "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" (UID: "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:04.699130 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.698740    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-data\") pod \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\" (UID: \"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a\") "
Oct 30 13:53:04.699130 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.698787    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-workload-socket" pod "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" (UID: "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:04.699130 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.699022    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-credential-socket" pod "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" (UID: "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:04.699804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.699180    5893 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-credential-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.699804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.699201    5893 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-workload-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.699804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.699200    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-data" pod "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" (UID: "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:04.699804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.699217    5893 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-workload-socket\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.699804 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.699217    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istiod-ca-cert" pod "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" (UID: "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:04.701202 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.701165    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-envoy" pod "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" (UID: "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:53:04.701500 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.701473    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-token" pod "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" (UID: "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:04.701814 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.701788    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-kube-api-access-8s95v" pod "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" (UID: "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a"). InnerVolumeSpecName "kube-api-access-8s95v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:04.701901 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.701840    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-podinfo" pod "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" (UID: "b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:53:04.799805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.799754    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s95v\" (UniqueName: \"kubernetes.io/projected/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-kube-api-access-8s95v\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.799805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.799803    5893 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istiod-ca-cert\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.799805 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.799813    5893 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-data\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.800056 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.799821    5893 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-podinfo\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.800056 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.799829    5893 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-token\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.800056 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.799836    5893 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a-istio-envoy\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:53:04.943686 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.943648    5893 generic.go:356] "Generic (PLEG): container finished" podID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerID="d986a4564eb26a4d07eeed7c5f571e304bdb9e13ea91aca6d16ba8aa88a810d3" exitCode=0
Oct 30 13:53:04.943876 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.943705    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"
Oct 30 13:53:04.943876 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.943730    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6" event={"ID":"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a","Type":"ContainerDied","Data":"d986a4564eb26a4d07eeed7c5f571e304bdb9e13ea91aca6d16ba8aa88a810d3"}
Oct 30 13:53:04.943876 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.943805    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6" event={"ID":"b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a","Type":"ContainerDied","Data":"fbbe6ae455dc97bb85a3554f8accd0ebdb10163b3281a3824e9b725e671d4ca0"}
Oct 30 13:53:04.943876 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.943835    5893 scope.go:122] "RemoveContainer" containerID="d986a4564eb26a4d07eeed7c5f571e304bdb9e13ea91aca6d16ba8aa88a810d3"
Oct 30 13:53:04.954138 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.954118    5893 scope.go:122] "RemoveContainer" containerID="46718dd9451cc266fbd615050b0a8c904b7fb714fe646c285f5ea89f61b1d5c4"
Oct 30 13:53:04.963368 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.963350    5893 scope.go:122] "RemoveContainer" containerID="d986a4564eb26a4d07eeed7c5f571e304bdb9e13ea91aca6d16ba8aa88a810d3"
Oct 30 13:53:04.963630 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:53:04.963610    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d986a4564eb26a4d07eeed7c5f571e304bdb9e13ea91aca6d16ba8aa88a810d3\": container with ID starting with d986a4564eb26a4d07eeed7c5f571e304bdb9e13ea91aca6d16ba8aa88a810d3 not found: ID does not exist" containerID="d986a4564eb26a4d07eeed7c5f571e304bdb9e13ea91aca6d16ba8aa88a810d3"
Oct 30 13:53:04.963679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.963641    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d986a4564eb26a4d07eeed7c5f571e304bdb9e13ea91aca6d16ba8aa88a810d3"} err="failed to get container status \"d986a4564eb26a4d07eeed7c5f571e304bdb9e13ea91aca6d16ba8aa88a810d3\": rpc error: code = NotFound desc = could not find container \"d986a4564eb26a4d07eeed7c5f571e304bdb9e13ea91aca6d16ba8aa88a810d3\": container with ID starting with d986a4564eb26a4d07eeed7c5f571e304bdb9e13ea91aca6d16ba8aa88a810d3 not found: ID does not exist"
Oct 30 13:53:04.963679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.963658    5893 scope.go:122] "RemoveContainer" containerID="46718dd9451cc266fbd615050b0a8c904b7fb714fe646c285f5ea89f61b1d5c4"
Oct 30 13:53:04.963900 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:53:04.963879    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46718dd9451cc266fbd615050b0a8c904b7fb714fe646c285f5ea89f61b1d5c4\": container with ID starting with 46718dd9451cc266fbd615050b0a8c904b7fb714fe646c285f5ea89f61b1d5c4 not found: ID does not exist" containerID="46718dd9451cc266fbd615050b0a8c904b7fb714fe646c285f5ea89f61b1d5c4"
Oct 30 13:53:04.963975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.963911    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46718dd9451cc266fbd615050b0a8c904b7fb714fe646c285f5ea89f61b1d5c4"} err="failed to get container status \"46718dd9451cc266fbd615050b0a8c904b7fb714fe646c285f5ea89f61b1d5c4\": rpc error: code = NotFound desc = could not find container \"46718dd9451cc266fbd615050b0a8c904b7fb714fe646c285f5ea89f61b1d5c4\": container with ID starting with 46718dd9451cc266fbd615050b0a8c904b7fb714fe646c285f5ea89f61b1d5c4 not found: ID does not exist"
Oct 30 13:53:04.989240 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.989203    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"]
Oct 30 13:53:04.998702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:04.998677    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-bf43af-openshift-default-6cf6fd9cbfkqdh6"]
Oct 30 13:53:05.556881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:05.556848    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k" podUID="ca4c5a90-e1f3-4723-a12b-ca85b8508221" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:53:05.556881 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:05.556876    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw" podUID="18ef5379-55bf-4488-9caf-f2bcf04cf941" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:53:06.532538 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:06.532499    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" path="/var/lib/kubelet/pods/b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a/volumes"
Oct 30 13:53:14.986349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:14.985203    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"
Oct 30 13:53:15.015997 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.015967    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"
Oct 30 13:53:15.023332 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.023299    5893 generic.go:356] "Generic (PLEG): container finished" podID="18ef5379-55bf-4488-9caf-f2bcf04cf941" containerID="39a3d300402bee60f437297e0a665838d25deaa141714188c4036704cf60b38c" exitCode=0
Oct 30 13:53:15.023580 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.023348    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw" event={"ID":"18ef5379-55bf-4488-9caf-f2bcf04cf941","Type":"ContainerDied","Data":"39a3d300402bee60f437297e0a665838d25deaa141714188c4036704cf60b38c"}
Oct 30 13:53:15.023718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.023663    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw" event={"ID":"18ef5379-55bf-4488-9caf-f2bcf04cf941","Type":"ContainerDied","Data":"c14477facd90389a60da1e3574598949091c9ace11857c64dbd6978fb6d01ea6"}
Oct 30 13:53:15.023718 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.023707    5893 scope.go:122] "RemoveContainer" containerID="39a3d300402bee60f437297e0a665838d25deaa141714188c4036704cf60b38c"
Oct 30 13:53:15.024861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.023385    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"
Oct 30 13:53:15.025353 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.025329    5893 generic.go:356] "Generic (PLEG): container finished" podID="ca4c5a90-e1f3-4723-a12b-ca85b8508221" containerID="48af575fa7a02e41734d06979bd42325957cdd375b3d63937a306bef3f62baeb" exitCode=0
Oct 30 13:53:15.025476 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.025401    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"
Oct 30 13:53:15.025531 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.025400    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k" event={"ID":"ca4c5a90-e1f3-4723-a12b-ca85b8508221","Type":"ContainerDied","Data":"48af575fa7a02e41734d06979bd42325957cdd375b3d63937a306bef3f62baeb"}
Oct 30 13:53:15.025574 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.025561    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k" event={"ID":"ca4c5a90-e1f3-4723-a12b-ca85b8508221","Type":"ContainerDied","Data":"a9aa5a85d906089dcb992571fed169e1243f716da0017862b4e5dc6875d39627"}
Oct 30 13:53:15.034959 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.034937    5893 scope.go:122] "RemoveContainer" containerID="39a3d300402bee60f437297e0a665838d25deaa141714188c4036704cf60b38c"
Oct 30 13:53:15.035253 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:53:15.035232    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39a3d300402bee60f437297e0a665838d25deaa141714188c4036704cf60b38c\": container with ID starting with 39a3d300402bee60f437297e0a665838d25deaa141714188c4036704cf60b38c not found: ID does not exist" containerID="39a3d300402bee60f437297e0a665838d25deaa141714188c4036704cf60b38c"
Oct 30 13:53:15.035358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.035262    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39a3d300402bee60f437297e0a665838d25deaa141714188c4036704cf60b38c"} err="failed to get container status \"39a3d300402bee60f437297e0a665838d25deaa141714188c4036704cf60b38c\": rpc error: code = NotFound desc = could not find container \"39a3d300402bee60f437297e0a665838d25deaa141714188c4036704cf60b38c\": container with ID starting with 39a3d300402bee60f437297e0a665838d25deaa141714188c4036704cf60b38c not found: ID does not exist"
Oct 30 13:53:15.035358 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.035305    5893 scope.go:122] "RemoveContainer" containerID="48af575fa7a02e41734d06979bd42325957cdd375b3d63937a306bef3f62baeb"
Oct 30 13:53:15.043433 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.043414    5893 scope.go:122] "RemoveContainer" containerID="48af575fa7a02e41734d06979bd42325957cdd375b3d63937a306bef3f62baeb"
Oct 30 13:53:15.043816 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:53:15.043774    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48af575fa7a02e41734d06979bd42325957cdd375b3d63937a306bef3f62baeb\": container with ID starting with 48af575fa7a02e41734d06979bd42325957cdd375b3d63937a306bef3f62baeb not found: ID does not exist" containerID="48af575fa7a02e41734d06979bd42325957cdd375b3d63937a306bef3f62baeb"
Oct 30 13:53:15.043919 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.043815    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48af575fa7a02e41734d06979bd42325957cdd375b3d63937a306bef3f62baeb"} err="failed to get container status \"48af575fa7a02e41734d06979bd42325957cdd375b3d63937a306bef3f62baeb\": rpc error: code = NotFound desc = could not find container \"48af575fa7a02e41734d06979bd42325957cdd375b3d63937a306bef3f62baeb\": container with ID starting with 48af575fa7a02e41734d06979bd42325957cdd375b3d63937a306bef3f62baeb not found: ID does not exist"
Oct 30 13:53:15.172936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.172907    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18ef5379-55bf-4488-9caf-f2bcf04cf941-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ef5379-55bf-4488-9caf-f2bcf04cf941-plugins-config-volume\") pod \"18ef5379-55bf-4488-9caf-f2bcf04cf941\" (UID: \"18ef5379-55bf-4488-9caf-f2bcf04cf941\") "
Oct 30 13:53:15.173115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.172949    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4c5a90-e1f3-4723-a12b-ca85b8508221-kube-api-access-k5b52\" (UniqueName: \"kubernetes.io/projected/ca4c5a90-e1f3-4723-a12b-ca85b8508221-kube-api-access-k5b52\") pod \"ca4c5a90-e1f3-4723-a12b-ca85b8508221\" (UID: \"ca4c5a90-e1f3-4723-a12b-ca85b8508221\") "
Oct 30 13:53:15.173115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.173013    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18ef5379-55bf-4488-9caf-f2bcf04cf941-kube-api-access-lqcxx\" (UniqueName: \"kubernetes.io/projected/18ef5379-55bf-4488-9caf-f2bcf04cf941-kube-api-access-lqcxx\") pod \"18ef5379-55bf-4488-9caf-f2bcf04cf941\" (UID: \"18ef5379-55bf-4488-9caf-f2bcf04cf941\") "
Oct 30 13:53:15.173115 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.173040    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca4c5a90-e1f3-4723-a12b-ca85b8508221-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4c5a90-e1f3-4723-a12b-ca85b8508221-plugins-config-volume\") pod \"ca4c5a90-e1f3-4723-a12b-ca85b8508221\" (UID: \"ca4c5a90-e1f3-4723-a12b-ca85b8508221\") "
Oct 30 13:53:15.173408 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.173382    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18ef5379-55bf-4488-9caf-f2bcf04cf941-plugins-config-volume" pod "18ef5379-55bf-4488-9caf-f2bcf04cf941" (UID: "18ef5379-55bf-4488-9caf-f2bcf04cf941"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:15.173529 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.173508    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4c5a90-e1f3-4723-a12b-ca85b8508221-plugins-config-volume" pod "ca4c5a90-e1f3-4723-a12b-ca85b8508221" (UID: "ca4c5a90-e1f3-4723-a12b-ca85b8508221"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:15.175381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.175358    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18ef5379-55bf-4488-9caf-f2bcf04cf941-kube-api-access-lqcxx" pod "18ef5379-55bf-4488-9caf-f2bcf04cf941" (UID: "18ef5379-55bf-4488-9caf-f2bcf04cf941"). InnerVolumeSpecName "kube-api-access-lqcxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:15.175524 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.175381    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4c5a90-e1f3-4723-a12b-ca85b8508221-kube-api-access-k5b52" pod "ca4c5a90-e1f3-4723-a12b-ca85b8508221" (UID: "ca4c5a90-e1f3-4723-a12b-ca85b8508221"). InnerVolumeSpecName "kube-api-access-k5b52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:15.274839 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.274802    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18ef5379-55bf-4488-9caf-f2bcf04cf941-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:53:15.274839 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.274833    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5b52\" (UniqueName: \"kubernetes.io/projected/ca4c5a90-e1f3-4723-a12b-ca85b8508221-kube-api-access-k5b52\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:53:15.274839 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.274842    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqcxx\" (UniqueName: \"kubernetes.io/projected/18ef5379-55bf-4488-9caf-f2bcf04cf941-kube-api-access-lqcxx\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:53:15.275074 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.274851    5893 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4c5a90-e1f3-4723-a12b-ca85b8508221-plugins-config-volume\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:53:15.367749 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.367713    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"]
Oct 30 13:53:15.377711 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.377682    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-bf43af-799d9bfl5k"]
Oct 30 13:53:15.406808 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.406778    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"]
Oct 30 13:53:15.410840 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:15.410808    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-bf43af-6f55d5df92mnxw"]
Oct 30 13:53:16.530247 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:16.530214    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18ef5379-55bf-4488-9caf-f2bcf04cf941" path="/var/lib/kubelet/pods/18ef5379-55bf-4488-9caf-f2bcf04cf941/volumes"
Oct 30 13:53:16.530679 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:16.530622    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca4c5a90-e1f3-4723-a12b-ca85b8508221" path="/var/lib/kubelet/pods/ca4c5a90-e1f3-4723-a12b-ca85b8508221/volumes"
Oct 30 13:53:18.644850 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:53:18.644812    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:53:22.148750 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.148709    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"]
Oct 30 13:53:22.149448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.149383    5893 cpu_manager.go:414] "containerMap: removing container" podUID="ca4c5a90-e1f3-4723-a12b-ca85b8508221" containerName="payload-pre-processing"
Oct 30 13:53:22.149448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.149404    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca4c5a90-e1f3-4723-a12b-ca85b8508221" containerName="payload-pre-processing"
Oct 30 13:53:22.149448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.149426    5893 cpu_manager.go:414] "containerMap: removing container" podUID="18ef5379-55bf-4488-9caf-f2bcf04cf941" containerName="payload-processing"
Oct 30 13:53:22.149448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.149433    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18ef5379-55bf-4488-9caf-f2bcf04cf941" containerName="payload-processing"
Oct 30 13:53:22.149448 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.149447    5893 cpu_manager.go:414] "containerMap: removing container" podUID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerName="istio-proxy"
Oct 30 13:53:22.149744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.149455    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerName="istio-proxy"
Oct 30 13:53:22.149744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.149466    5893 cpu_manager.go:414] "containerMap: removing container" podUID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerName="istio-proxy"
Oct 30 13:53:22.149744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.149473    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerName="istio-proxy"
Oct 30 13:53:22.149744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.149585    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca4c5a90-e1f3-4723-a12b-ca85b8508221" containerName="payload-pre-processing"
Oct 30 13:53:22.149744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.149602    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="18ef5379-55bf-4488-9caf-f2bcf04cf941" containerName="payload-processing"
Oct 30 13:53:22.149744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.149614    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerName="istio-proxy"
Oct 30 13:53:22.149744 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.149624    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="b84f1ca9-009d-4c67-af4c-6cbd9c13ef0a" containerName="istio-proxy"
Oct 30 13:53:22.152829 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.152804    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.175413 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.175374    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-8959c618-kserve-self-signed-certs\""
Oct 30 13:53:22.177152 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.177127    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"]
Oct 30 13:53:22.243208 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.243174    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7fxg\" (UniqueName: \"kubernetes.io/projected/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-kube-api-access-w7fxg\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.243407 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.243213    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-home\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.243407 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.243233    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-tmp-dir\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.243407 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.243336    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-tls-certs\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.243523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.243437    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-model-cache\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.243523 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.243473    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-dshm\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.344587 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.344544    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-home\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.344587 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.344593    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-tmp-dir\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.344861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.344629    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-tls-certs\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.344861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.344723    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-model-cache\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.344861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.344768    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-dshm\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.344861 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.344844    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7fxg\" (UniqueName: \"kubernetes.io/projected/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-kube-api-access-w7fxg\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.345073 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.345033    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-home\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.345359 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.345328    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-tmp-dir\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.345677 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.345425    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-model-cache\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.347303 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.347257    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-dshm\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.347713 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.347692    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-tls-certs\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.374329 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.368231    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7fxg\" (UniqueName: \"kubernetes.io/projected/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-kube-api-access-w7fxg\") pod \"e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") " pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.468212 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.468180    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:22.770029 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:22.769930    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"]
Oct 30 13:53:22.774638 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:53:22.774607    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a156ba6_46a9_4c2f_90df_df9565e0d6b1.slice/crio-6805ee49de583474e63ab75f19116460f60ab860a47194c84c79227b36b51db1 WatchSource:0}: Error finding container 6805ee49de583474e63ab75f19116460f60ab860a47194c84c79227b36b51db1: Status 404 returned error can't find the container with id 6805ee49de583474e63ab75f19116460f60ab860a47194c84c79227b36b51db1
Oct 30 13:53:23.069615 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:23.069525    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z" event={"ID":"3a156ba6-46a9-4c2f-90df-df9565e0d6b1","Type":"ContainerStarted","Data":"458b1562518848f282126579a028de8e6764486b5c2b50fe625b45a22a910c9d"}
Oct 30 13:53:23.069615 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:23.069571    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z" event={"ID":"3a156ba6-46a9-4c2f-90df-df9565e0d6b1","Type":"ContainerStarted","Data":"6805ee49de583474e63ab75f19116460f60ab860a47194c84c79227b36b51db1"}
Oct 30 13:53:23.110251 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:23.110183    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z" podStartSLOduration=1.110165423 podStartE2EDuration="1.110165423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:23.108071908 +0000 UTC m=+4044.691501620" watchObservedRunningTime="2026-10-30 13:53:23.110165423 +0000 UTC m=+4044.693595120"
Oct 30 13:53:23.468437 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:23.468408    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:32.468335 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:32.468300    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:32.481129 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:32.481108    5893 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:33.148177 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:33.148150    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:53:48.645545 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:53:48.645463    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:53:53.688931 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:53:53.688890    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"]
Oct 30 13:54:08.927743 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:08.927685    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z" podUID="3a156ba6-46a9-4c2f-90df-df9565e0d6b1" containerName="main" containerID="cri-o://458b1562518848f282126579a028de8e6764486b5c2b50fe625b45a22a910c9d" gracePeriod=45
Oct 30 13:54:09.182339 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.182258    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:54:09.251602 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.251561    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-model-cache\") pod \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") "
Oct 30 13:54:09.251827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.251637    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-kube-api-access-w7fxg\" (UniqueName: \"kubernetes.io/projected/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-kube-api-access-w7fxg\") pod \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") "
Oct 30 13:54:09.251827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.251698    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-tmp-dir\") pod \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") "
Oct 30 13:54:09.251827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.251726    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-tls-certs\") pod \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") "
Oct 30 13:54:09.251827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.251751    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-dshm\") pod \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") "
Oct 30 13:54:09.251827 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.251795    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-home\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-home\") pod \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\" (UID: \"3a156ba6-46a9-4c2f-90df-df9565e0d6b1\") "
Oct 30 13:54:09.252190 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.251867    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-model-cache" pod "3a156ba6-46a9-4c2f-90df-df9565e0d6b1" (UID: "3a156ba6-46a9-4c2f-90df-df9565e0d6b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:09.252190 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.252032    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-tmp-dir" pod "3a156ba6-46a9-4c2f-90df-df9565e0d6b1" (UID: "3a156ba6-46a9-4c2f-90df-df9565e0d6b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:09.252374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.252230    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-home" pod "3a156ba6-46a9-4c2f-90df-df9565e0d6b1" (UID: "3a156ba6-46a9-4c2f-90df-df9565e0d6b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:09.252374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.252231    5893 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-tmp-dir\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.252374 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.252295    5893 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-model-cache\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.254221 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.254198    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-kube-api-access-w7fxg" pod "3a156ba6-46a9-4c2f-90df-df9565e0d6b1" (UID: "3a156ba6-46a9-4c2f-90df-df9565e0d6b1"). InnerVolumeSpecName "kube-api-access-w7fxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:09.254490 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.254472    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-tls-certs" pod "3a156ba6-46a9-4c2f-90df-df9565e0d6b1" (UID: "3a156ba6-46a9-4c2f-90df-df9565e0d6b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:09.254568 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.254554    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-dshm" pod "3a156ba6-46a9-4c2f-90df-df9565e0d6b1" (UID: "3a156ba6-46a9-4c2f-90df-df9565e0d6b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:09.353443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.353408    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7fxg\" (UniqueName: \"kubernetes.io/projected/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-kube-api-access-w7fxg\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.353443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.353436    5893 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-tls-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.353443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.353446    5893 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-dshm\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.353443 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.353455    5893 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a156ba6-46a9-4c2f-90df-df9565e0d6b1-home\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.390640 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.390605    5893 generic.go:356] "Generic (PLEG): container finished" podID="3a156ba6-46a9-4c2f-90df-df9565e0d6b1" containerID="458b1562518848f282126579a028de8e6764486b5c2b50fe625b45a22a910c9d" exitCode=0
Oct 30 13:54:09.390847 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.390648    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"
Oct 30 13:54:09.390847 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.390701    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z" event={"ID":"3a156ba6-46a9-4c2f-90df-df9565e0d6b1","Type":"ContainerDied","Data":"458b1562518848f282126579a028de8e6764486b5c2b50fe625b45a22a910c9d"}
Oct 30 13:54:09.390847 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.390747    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z" event={"ID":"3a156ba6-46a9-4c2f-90df-df9565e0d6b1","Type":"ContainerDied","Data":"6805ee49de583474e63ab75f19116460f60ab860a47194c84c79227b36b51db1"}
Oct 30 13:54:09.390847 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.390771    5893 scope.go:122] "RemoveContainer" containerID="458b1562518848f282126579a028de8e6764486b5c2b50fe625b45a22a910c9d"
Oct 30 13:54:09.400638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.400621    5893 scope.go:122] "RemoveContainer" containerID="458b1562518848f282126579a028de8e6764486b5c2b50fe625b45a22a910c9d"
Oct 30 13:54:09.400965 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:54:09.400947    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"458b1562518848f282126579a028de8e6764486b5c2b50fe625b45a22a910c9d\": container with ID starting with 458b1562518848f282126579a028de8e6764486b5c2b50fe625b45a22a910c9d not found: ID does not exist" containerID="458b1562518848f282126579a028de8e6764486b5c2b50fe625b45a22a910c9d"
Oct 30 13:54:09.401024 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.400977    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"458b1562518848f282126579a028de8e6764486b5c2b50fe625b45a22a910c9d"} err="failed to get container status \"458b1562518848f282126579a028de8e6764486b5c2b50fe625b45a22a910c9d\": rpc error: code = NotFound desc = could not find container \"458b1562518848f282126579a028de8e6764486b5c2b50fe625b45a22a910c9d\": container with ID starting with 458b1562518848f282126579a028de8e6764486b5c2b50fe625b45a22a910c9d not found: ID does not exist"
Oct 30 13:54:09.433902 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.433820    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"]
Oct 30 13:54:09.438833 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:09.438809    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z"]
Oct 30 13:54:10.148435 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:10.137677    5893 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-8959c618-kserve-58bd865d6f-bn87z" podUID="3a156ba6-46a9-4c2f-90df-df9565e0d6b1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.226:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:54:10.529632 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:10.529600    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a156ba6-46a9-4c2f-90df-df9565e0d6b1" path="/var/lib/kubelet/pods/3a156ba6-46a9-4c2f-90df-df9565e0d6b1/volumes"
Oct 30 13:54:18.644939 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:54:18.644902    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:54:30.650317 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.650183    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"]
Oct 30 13:54:30.650995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.650900    5893 cpu_manager.go:414] "containerMap: removing container" podUID="3a156ba6-46a9-4c2f-90df-df9565e0d6b1" containerName="main"
Oct 30 13:54:30.650995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.650917    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a156ba6-46a9-4c2f-90df-df9565e0d6b1" containerName="main"
Oct 30 13:54:30.651180 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.651015    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a156ba6-46a9-4c2f-90df-df9565e0d6b1" containerName="main"
Oct 30 13:54:30.655544 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.655520    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:30.683177 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.683145    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"]
Oct 30 13:54:30.705081 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.705036    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605511ac-b45e-4cdd-bcf1-d2357e70ce5e-metrics-certs\") pod \"maas-api-595cc648c-rv8dk\" (UID: \"605511ac-b45e-4cdd-bcf1-d2357e70ce5e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:30.705349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.705222    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/605511ac-b45e-4cdd-bcf1-d2357e70ce5e-openshift-service-ca\") pod \"maas-api-595cc648c-rv8dk\" (UID: \"605511ac-b45e-4cdd-bcf1-d2357e70ce5e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:30.705349 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.705328    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5f84\" (UniqueName: \"kubernetes.io/projected/605511ac-b45e-4cdd-bcf1-d2357e70ce5e-kube-api-access-c5f84\") pod \"maas-api-595cc648c-rv8dk\" (UID: \"605511ac-b45e-4cdd-bcf1-d2357e70ce5e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:30.705510 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.705486    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/605511ac-b45e-4cdd-bcf1-d2357e70ce5e-maas-api-tls\") pod \"maas-api-595cc648c-rv8dk\" (UID: \"605511ac-b45e-4cdd-bcf1-d2357e70ce5e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:30.806479 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.806438    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/605511ac-b45e-4cdd-bcf1-d2357e70ce5e-maas-api-tls\") pod \"maas-api-595cc648c-rv8dk\" (UID: \"605511ac-b45e-4cdd-bcf1-d2357e70ce5e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:30.806479 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.806482    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605511ac-b45e-4cdd-bcf1-d2357e70ce5e-metrics-certs\") pod \"maas-api-595cc648c-rv8dk\" (UID: \"605511ac-b45e-4cdd-bcf1-d2357e70ce5e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:30.806762 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.806538    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/605511ac-b45e-4cdd-bcf1-d2357e70ce5e-openshift-service-ca\") pod \"maas-api-595cc648c-rv8dk\" (UID: \"605511ac-b45e-4cdd-bcf1-d2357e70ce5e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:30.806762 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.806570    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5f84\" (UniqueName: \"kubernetes.io/projected/605511ac-b45e-4cdd-bcf1-d2357e70ce5e-kube-api-access-c5f84\") pod \"maas-api-595cc648c-rv8dk\" (UID: \"605511ac-b45e-4cdd-bcf1-d2357e70ce5e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:30.807620 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.807593    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/605511ac-b45e-4cdd-bcf1-d2357e70ce5e-openshift-service-ca\") pod \"maas-api-595cc648c-rv8dk\" (UID: \"605511ac-b45e-4cdd-bcf1-d2357e70ce5e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:30.809301 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.809254    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/605511ac-b45e-4cdd-bcf1-d2357e70ce5e-metrics-certs\") pod \"maas-api-595cc648c-rv8dk\" (UID: \"605511ac-b45e-4cdd-bcf1-d2357e70ce5e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:30.809423 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.809397    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/605511ac-b45e-4cdd-bcf1-d2357e70ce5e-maas-api-tls\") pod \"maas-api-595cc648c-rv8dk\" (UID: \"605511ac-b45e-4cdd-bcf1-d2357e70ce5e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:30.827593 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.827572    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5f84\" (UniqueName: \"kubernetes.io/projected/605511ac-b45e-4cdd-bcf1-d2357e70ce5e-kube-api-access-c5f84\") pod \"maas-api-595cc648c-rv8dk\" (UID: \"605511ac-b45e-4cdd-bcf1-d2357e70ce5e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:30.971305 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:30.971263    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:31.272291 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:31.272197    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"]
Oct 30 13:54:31.277534 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:54:31.277503    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605511ac_b45e_4cdd_bcf1_d2357e70ce5e.slice/crio-9e695674e948f722236e775690078e2ac0c529cc7f29380e6037f316ec53a2ac WatchSource:0}: Error finding container 9e695674e948f722236e775690078e2ac0c529cc7f29380e6037f316ec53a2ac: Status 404 returned error can't find the container with id 9e695674e948f722236e775690078e2ac0c529cc7f29380e6037f316ec53a2ac
Oct 30 13:54:31.278887 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:31.278869    5893 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:54:31.512491 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:31.512448    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk" event={"ID":"605511ac-b45e-4cdd-bcf1-d2357e70ce5e","Type":"ContainerStarted","Data":"9e695674e948f722236e775690078e2ac0c529cc7f29380e6037f316ec53a2ac"}
Oct 30 13:54:32.520288 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:32.520245    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk" event={"ID":"605511ac-b45e-4cdd-bcf1-d2357e70ce5e","Type":"ContainerStarted","Data":"9a0c37655ee42cb2991d3daa6943730063633c71f1c3e2169ef4fc1e5b81d04c"}
Oct 30 13:54:32.520791 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:32.520450    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:32.563698 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:32.563635    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk" podStartSLOduration=2.250455022 podStartE2EDuration="2.563615864s" totalImagesPullingTime="313.160842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:32.562151899 +0000 UTC m=+4114.145581610" watchObservedRunningTime="2026-10-30 13:54:32.563615864 +0000 UTC m=+4114.147045558"
Oct 30 13:54:38.531297 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:38.531241    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-rv8dk"
Oct 30 13:54:38.602638 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:38.602601    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"]
Oct 30 13:54:38.602887 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:38.602853    5893 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp" podUID="4380a330-1cdb-42b1-ab66-50f9b6595b84" containerName="maas-api" containerID="cri-o://a4f91eef0cd582c4b4dea864971e42860808a87abebf6ecce064b3de907075a2" gracePeriod=30
Oct 30 13:54:39.882755 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:39.882725    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 13:54:40.004110 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.004067    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4380a330-1cdb-42b1-ab66-50f9b6595b84-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4380a330-1cdb-42b1-ab66-50f9b6595b84-maas-api-tls\") pod \"4380a330-1cdb-42b1-ab66-50f9b6595b84\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") "
Oct 30 13:54:40.004355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.004215    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4380a330-1cdb-42b1-ab66-50f9b6595b84-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4380a330-1cdb-42b1-ab66-50f9b6595b84-metrics-certs\") pod \"4380a330-1cdb-42b1-ab66-50f9b6595b84\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") "
Oct 30 13:54:40.004355 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.004330    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4380a330-1cdb-42b1-ab66-50f9b6595b84-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4380a330-1cdb-42b1-ab66-50f9b6595b84-openshift-service-ca\") pod \"4380a330-1cdb-42b1-ab66-50f9b6595b84\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") "
Oct 30 13:54:40.004516 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.004374    5893 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4380a330-1cdb-42b1-ab66-50f9b6595b84-kube-api-access-5f948\" (UniqueName: \"kubernetes.io/projected/4380a330-1cdb-42b1-ab66-50f9b6595b84-kube-api-access-5f948\") pod \"4380a330-1cdb-42b1-ab66-50f9b6595b84\" (UID: \"4380a330-1cdb-42b1-ab66-50f9b6595b84\") "
Oct 30 13:54:40.004768 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.004739    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4380a330-1cdb-42b1-ab66-50f9b6595b84-openshift-service-ca" pod "4380a330-1cdb-42b1-ab66-50f9b6595b84" (UID: "4380a330-1cdb-42b1-ab66-50f9b6595b84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:40.007438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.007407    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4380a330-1cdb-42b1-ab66-50f9b6595b84-maas-api-tls" pod "4380a330-1cdb-42b1-ab66-50f9b6595b84" (UID: "4380a330-1cdb-42b1-ab66-50f9b6595b84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:40.019453 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.019422    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4380a330-1cdb-42b1-ab66-50f9b6595b84-kube-api-access-5f948" pod "4380a330-1cdb-42b1-ab66-50f9b6595b84" (UID: "4380a330-1cdb-42b1-ab66-50f9b6595b84"). InnerVolumeSpecName "kube-api-access-5f948". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:40.025586 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.025555    5893 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4380a330-1cdb-42b1-ab66-50f9b6595b84-metrics-certs" pod "4380a330-1cdb-42b1-ab66-50f9b6595b84" (UID: "4380a330-1cdb-42b1-ab66-50f9b6595b84"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:54:40.105788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.105743    5893 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f948\" (UniqueName: \"kubernetes.io/projected/4380a330-1cdb-42b1-ab66-50f9b6595b84-kube-api-access-5f948\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:54:40.105788 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.105788    5893 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4380a330-1cdb-42b1-ab66-50f9b6595b84-maas-api-tls\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:54:40.106105 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.105804    5893 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4380a330-1cdb-42b1-ab66-50f9b6595b84-metrics-certs\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:54:40.106105 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.105818    5893 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4380a330-1cdb-42b1-ab66-50f9b6595b84-openshift-service-ca\") on node \"ip-10-0-141-232.ec2.internal\" DevicePath \"\""
Oct 30 13:54:40.564323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.564250    5893 generic.go:356] "Generic (PLEG): container finished" podID="4380a330-1cdb-42b1-ab66-50f9b6595b84" containerID="a4f91eef0cd582c4b4dea864971e42860808a87abebf6ecce064b3de907075a2" exitCode=0
Oct 30 13:54:40.564995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.564565    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp" event={"ID":"4380a330-1cdb-42b1-ab66-50f9b6595b84","Type":"ContainerDied","Data":"a4f91eef0cd582c4b4dea864971e42860808a87abebf6ecce064b3de907075a2"}
Oct 30 13:54:40.564995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.564640    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp" event={"ID":"4380a330-1cdb-42b1-ab66-50f9b6595b84","Type":"ContainerDied","Data":"912386c571855d07f460801424b6ea9d2ac2ba83bf3b6abeaceb878e35a6fbd7"}
Oct 30 13:54:40.564995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.564673    5893 scope.go:122] "RemoveContainer" containerID="a4f91eef0cd582c4b4dea864971e42860808a87abebf6ecce064b3de907075a2"
Oct 30 13:54:40.564995 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.564818    5893 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"
Oct 30 13:54:40.583905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.583881    5893 scope.go:122] "RemoveContainer" containerID="a4f91eef0cd582c4b4dea864971e42860808a87abebf6ecce064b3de907075a2"
Oct 30 13:54:40.584465 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:54:40.584431    5893 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f91eef0cd582c4b4dea864971e42860808a87abebf6ecce064b3de907075a2\": container with ID starting with a4f91eef0cd582c4b4dea864971e42860808a87abebf6ecce064b3de907075a2 not found: ID does not exist" containerID="a4f91eef0cd582c4b4dea864971e42860808a87abebf6ecce064b3de907075a2"
Oct 30 13:54:40.584595 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.584499    5893 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f91eef0cd582c4b4dea864971e42860808a87abebf6ecce064b3de907075a2"} err="failed to get container status \"a4f91eef0cd582c4b4dea864971e42860808a87abebf6ecce064b3de907075a2\": rpc error: code = NotFound desc = could not find container \"a4f91eef0cd582c4b4dea864971e42860808a87abebf6ecce064b3de907075a2\": container with ID starting with a4f91eef0cd582c4b4dea864971e42860808a87abebf6ecce064b3de907075a2 not found: ID does not exist"
Oct 30 13:54:40.621725 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.621695    5893 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"]
Oct 30 13:54:40.630773 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:40.630747    5893 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-lr5lp"]
Oct 30 13:54:42.530229 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:54:42.530194    5893 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4380a330-1cdb-42b1-ab66-50f9b6595b84" path="/var/lib/kubelet/pods/4380a330-1cdb-42b1-ab66-50f9b6595b84/volumes"
Oct 30 13:54:48.668455 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:54:48.668404    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:55:18.644368 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:55:18.644316    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:55:28.730738 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:28.730711    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-845dc99f6f-cc7nr_f1a839f3-745c-4270-a3ff-0145e0cd0ebc/authorino/0.log"
Oct 30 13:55:39.812842 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:39.812807    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-69c996b8fd-7phsv_dca88d7b-ad83-4b4c-bded-ced88f047b8b/manager/0.log"
Oct 30 13:55:41.338879 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:41.338849    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-gpsxn_3b1c3888-ac99-45cd-b338-c0b8418ccec2/manager/0.log"
Oct 30 13:55:42.347662 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:42.347630    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c4494f88b-zt6hf_ad9763b3-b806-4dc2-a756-b3ac03f747c8/copy-manifests/0.log"
Oct 30 13:55:42.381625 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:42.381593    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c4494f88b-zt6hf_ad9763b3-b806-4dc2-a756-b3ac03f747c8/manager/0.log"
Oct 30 13:55:43.639897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:43.639869    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-rv8dk_605511ac-b45e-4cdd-bcf1-d2357e70ce5e/maas-api/0.log"
Oct 30 13:55:47.285500 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:47.285472    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-845dc99f6f-cc7nr_f1a839f3-745c-4270-a3ff-0145e0cd0ebc/authorino/0.log"
Oct 30 13:55:48.203543 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:48.203505    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-kc2vn_446440ca-255b-4262-8c8f-76953e77f832/manager/0.log"
Oct 30 13:55:48.644998 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:55:48.644961    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:55:49.914428 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:49.914398    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cbd4cb4f6-7pqlr_82007c50-3dfd-457b-9e0c-8d69f0481a9f/kube-auth-proxy/0.log"
Oct 30 13:55:51.572351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:51.572300    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-jl8n2_742ed464-ac90-468a-92f7-edde51ba02bb/istio-proxy/0.log"
Oct 30 13:55:51.791381 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:51.791355    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-ktgn6_c66344b3-3175-40c0-b8a3-51ccf36bdc4f/istio-proxy/0.log"
Oct 30 13:55:52.202967 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:52.202939    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-ndplc_f49c9da9-a8bc-4139-bcc9-617e5a360704/istio-proxy/0.log"
Oct 30 13:55:54.275709 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:54.275681    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-2cmc8_387d629f-47f2-40b9-85f5-ff8d42310c88/main/0.log"
Oct 30 13:55:54.485975 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:54.485937    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-f94xm_d9296ecf-4e97-4ddb-9ef4-9d136b7317c9/main/0.log"
Oct 30 13:55:54.894702 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:54.894674    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65_66ed2849-eb53-426f-b0c5-5e059453882a/main/0.log"
Oct 30 13:55:55.105570 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:55.105544    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd_e2b6f77b-f0bc-4e73-9572-2e652b9f58ba/main/0.log"
Oct 30 13:55:56.040187 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:55:56.040151    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-845dc99f6f-cc7nr_f1a839f3-745c-4270-a3ff-0145e0cd0ebc/authorino/0.log"
Oct 30 13:56:01.502438 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:01.502409    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:56:01.514564 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:01.514542    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:56:01.527745 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:01.527718    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:56:01.539865 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:01.539842    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-dg4xp_f6897566-da42-4b8d-ba85-7ec304624f2b/ovn-acl-logging/0.log"
Oct 30 13:56:05.633789 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:05.633745    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-69c996b8fd-7phsv_dca88d7b-ad83-4b4c-bded-ced88f047b8b/manager/0.log"
Oct 30 13:56:06.964457 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:06.964418    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-gpsxn_3b1c3888-ac99-45cd-b338-c0b8418ccec2/manager/0.log"
Oct 30 13:56:07.947465 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:07.947442    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c4494f88b-zt6hf_ad9763b3-b806-4dc2-a756-b3ac03f747c8/copy-manifests/0.log"
Oct 30 13:56:07.978429 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:07.978401    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c4494f88b-zt6hf_ad9763b3-b806-4dc2-a756-b3ac03f747c8/manager/0.log"
Oct 30 13:56:08.997567 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:08.997519    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-rv8dk_605511ac-b45e-4cdd-bcf1-d2357e70ce5e/maas-api/0.log"
Oct 30 13:56:12.221072 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:12.215140    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-845dc99f6f-cc7nr_f1a839f3-745c-4270-a3ff-0145e0cd0ebc/authorino/0.log"
Oct 30 13:56:13.132243 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:13.132216    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-kc2vn_446440ca-255b-4262-8c8f-76953e77f832/manager/0.log"
Oct 30 13:56:14.654007 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:14.653974    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cbd4cb4f6-7pqlr_82007c50-3dfd-457b-9e0c-8d69f0481a9f/kube-auth-proxy/0.log"
Oct 30 13:56:16.297337 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:16.297311    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-jl8n2_742ed464-ac90-468a-92f7-edde51ba02bb/istio-proxy/0.log"
Oct 30 13:56:16.505752 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:16.505727    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-ktgn6_c66344b3-3175-40c0-b8a3-51ccf36bdc4f/istio-proxy/0.log"
Oct 30 13:56:16.908182 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:16.908152    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-ndplc_f49c9da9-a8bc-4139-bcc9-617e5a360704/istio-proxy/0.log"
Oct 30 13:56:18.512373 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:18.512346    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-2cmc8_387d629f-47f2-40b9-85f5-ff8d42310c88/main/0.log"
Oct 30 13:56:18.644562 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:56:18.644527    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:56:18.718451 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:18.718421    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-f94xm_d9296ecf-4e97-4ddb-9ef4-9d136b7317c9/main/0.log"
Oct 30 13:56:19.132247 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:19.132219    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-kgw65_66ed2849-eb53-426f-b0c5-5e059453882a/main/0.log"
Oct 30 13:56:19.327224 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:56:19.327198    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-65hcd_e2b6f77b-f0bc-4e73-9572-2e652b9f58ba/main/0.log"
Oct 30 13:56:48.645620 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:56:48.645526    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:57:18.645362 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:57:18.645314    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:57:43.910950 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:43.910916    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-79f2f_408c5c75-c039-4dc1-89c7-6b0f4412aa59/global-pull-secret-syncer/0.log"
Oct 30 13:57:45.063873 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:45.063839    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fd9xb_d08a5487-7eaa-4438-8fcd-3ead8dc6c20e/konnectivity-agent/0.log"
Oct 30 13:57:45.315228 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:45.315152    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-232.ec2.internal_6886c36025c26ee79576270d5200a014/haproxy/0.log"
Oct 30 13:57:48.644715 ip-10-0-141-232 kubenswrapper[5893]: E1030 13:57:48.644681    5893 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-k47jk" containerName="extractor"
Oct 30 13:57:51.551318 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:51.551216    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-845dc99f6f-cc7nr_f1a839f3-745c-4270-a3ff-0145e0cd0ebc/authorino/0.log"
Oct 30 13:57:52.135518 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:52.135484    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-57bb9bfff-kc2vn_446440ca-255b-4262-8c8f-76953e77f832/manager/0.log"
Oct 30 13:57:53.608635 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.608602    5893 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"]
Oct 30 13:57:53.609083 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.609012    5893 cpu_manager.go:414] "containerMap: removing container" podUID="4380a330-1cdb-42b1-ab66-50f9b6595b84" containerName="maas-api"
Oct 30 13:57:53.609083 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.609022    5893 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4380a330-1cdb-42b1-ab66-50f9b6595b84" containerName="maas-api"
Oct 30 13:57:53.609159 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.609129    5893 memory_manager.go:365] "RemoveStaleState removing state" podUID="4380a330-1cdb-42b1-ab66-50f9b6595b84" containerName="maas-api"
Oct 30 13:57:53.612790 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.612768    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.627488 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.627471    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cz4kh\"/\"openshift-service-ca.crt\""
Oct 30 13:57:53.627606 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.627529    5893 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cz4kh\"/\"kube-root-ca.crt\""
Oct 30 13:57:53.638706 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.638684    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"]
Oct 30 13:57:53.639886 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.639867    5893 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cz4kh\"/\"default-dockercfg-2bzw5\""
Oct 30 13:57:53.658942 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.658913    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58283e2c-5bb5-4bc3-895e-06654a003e7d-lib-modules\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.659118 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.658953    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58283e2c-5bb5-4bc3-895e-06654a003e7d-proc\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.659118 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.659019    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7wbx\" (UniqueName: \"kubernetes.io/projected/58283e2c-5bb5-4bc3-895e-06654a003e7d-kube-api-access-d7wbx\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.659118 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.659039    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58283e2c-5bb5-4bc3-895e-06654a003e7d-podres\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.659238 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.659127    5893 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58283e2c-5bb5-4bc3-895e-06654a003e7d-sys\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.759828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.759791    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7wbx\" (UniqueName: \"kubernetes.io/projected/58283e2c-5bb5-4bc3-895e-06654a003e7d-kube-api-access-d7wbx\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.760019 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.759838    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58283e2c-5bb5-4bc3-895e-06654a003e7d-podres\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.760019 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.759882    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58283e2c-5bb5-4bc3-895e-06654a003e7d-sys\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.760019 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.759980    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58283e2c-5bb5-4bc3-895e-06654a003e7d-lib-modules\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.760019 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.759988    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/58283e2c-5bb5-4bc3-895e-06654a003e7d-podres\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.760227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.760025    5893 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58283e2c-5bb5-4bc3-895e-06654a003e7d-proc\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.760227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.760028    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/58283e2c-5bb5-4bc3-895e-06654a003e7d-sys\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.760227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.760067    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/58283e2c-5bb5-4bc3-895e-06654a003e7d-proc\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.760227 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.760079    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/58283e2c-5bb5-4bc3-895e-06654a003e7d-lib-modules\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.781430 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.781401    5893 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7wbx\" (UniqueName: \"kubernetes.io/projected/58283e2c-5bb5-4bc3-895e-06654a003e7d-kube-api-access-d7wbx\") pod \"perf-node-gather-daemonset-phhb4\" (UID: \"58283e2c-5bb5-4bc3-895e-06654a003e7d\") " pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.827051 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.827020    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d43fce5b-ab9d-4750-bd9c-ae0c7e829d78/alertmanager/0.log"
Oct 30 13:57:53.925542 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.925434    5893 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:53.939471 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:53.939431    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d43fce5b-ab9d-4750-bd9c-ae0c7e829d78/config-reloader/0.log"
Oct 30 13:57:54.051889 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:54.051864    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d43fce5b-ab9d-4750-bd9c-ae0c7e829d78/kube-rbac-proxy-web/0.log"
Oct 30 13:57:54.170045 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:54.170021    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d43fce5b-ab9d-4750-bd9c-ae0c7e829d78/kube-rbac-proxy/0.log"
Oct 30 13:57:54.226970 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:54.226939    5893 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"]
Oct 30 13:57:54.230968 ip-10-0-141-232 kubenswrapper[5893]: W1030 13:57:54.230939    5893 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod58283e2c_5bb5_4bc3_895e_06654a003e7d.slice/crio-4a3672229e1118879aa718c3a346ec618495410f6dc76e86775f79b1750fda7a WatchSource:0}: Error finding container 4a3672229e1118879aa718c3a346ec618495410f6dc76e86775f79b1750fda7a: Status 404 returned error can't find the container with id 4a3672229e1118879aa718c3a346ec618495410f6dc76e86775f79b1750fda7a
Oct 30 13:57:54.285206 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:54.285184    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d43fce5b-ab9d-4750-bd9c-ae0c7e829d78/kube-rbac-proxy-metric/0.log"
Oct 30 13:57:54.398669 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:54.398641    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d43fce5b-ab9d-4750-bd9c-ae0c7e829d78/prom-label-proxy/0.log"
Oct 30 13:57:54.513761 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:54.513685    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d43fce5b-ab9d-4750-bd9c-ae0c7e829d78/init-config-reloader/0.log"
Oct 30 13:57:54.545897 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:54.545864    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4" event={"ID":"58283e2c-5bb5-4bc3-895e-06654a003e7d","Type":"ContainerStarted","Data":"0582f3518053859cc95ed7b4973a43dbf0d67c461c0ca1139a834ee61631372c"}
Oct 30 13:57:54.546068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:54.545911    5893 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4" event={"ID":"58283e2c-5bb5-4bc3-895e-06654a003e7d","Type":"ContainerStarted","Data":"4a3672229e1118879aa718c3a346ec618495410f6dc76e86775f79b1750fda7a"}
Oct 30 13:57:54.546068 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:54.546005    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:57:54.584983 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:54.584935    5893 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4" podStartSLOduration=1.584922103 podStartE2EDuration="1.584922103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:57:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:57:54.582421416 +0000 UTC m=+4316.165851109" watchObservedRunningTime="2026-10-30 13:57:54.584922103 +0000 UTC m=+4316.168351797"
Oct 30 13:57:54.751308 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:54.751260    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fkq8t_9b17184b-71db-4557-ae32-062ae53a9d53/kube-state-metrics/0.log"
Oct 30 13:57:54.856936 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:54.856861    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fkq8t_9b17184b-71db-4557-ae32-062ae53a9d53/kube-rbac-proxy-main/0.log"
Oct 30 13:57:54.974515 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:54.974490    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-fkq8t_9b17184b-71db-4557-ae32-062ae53a9d53/kube-rbac-proxy-self/0.log"
Oct 30 13:57:55.095323 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:55.095298    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-b6fc5f989-jm9qh_963f75f5-2539-456a-ae78-267d5b002bab/metrics-server/0.log"
Oct 30 13:57:55.210477 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:55.210395    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-krk2q_35840c58-b7a6-438e-aed5-6c0d9764a793/monitoring-plugin/0.log"
Oct 30 13:57:55.329737 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:55.329710    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2xmlk_afb494b7-75a0-400e-be16-74d8d6882460/node-exporter/0.log"
Oct 30 13:57:55.441834 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:55.441807    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2xmlk_afb494b7-75a0-400e-be16-74d8d6882460/kube-rbac-proxy/0.log"
Oct 30 13:57:55.557573 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:55.557547    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2xmlk_afb494b7-75a0-400e-be16-74d8d6882460/init-textfile/0.log"
Oct 30 13:57:56.431079 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:56.431050    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-p6cmw_89288ada-24c5-46af-ad86-7d21935ca939/kube-rbac-proxy-main/0.log"
Oct 30 13:57:56.548680 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:56.548654    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-p6cmw_89288ada-24c5-46af-ad86-7d21935ca939/kube-rbac-proxy-self/0.log"
Oct 30 13:57:56.665084 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:56.665054    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-p6cmw_89288ada-24c5-46af-ad86-7d21935ca939/openshift-state-metrics/0.log"
Oct 30 13:57:56.794327 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:56.794300    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/0.log"
Oct 30 13:57:56.795351 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:56.795335    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/prometheus/1.log"
Oct 30 13:57:56.882522 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:56.882494    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/config-reloader/0.log"
Oct 30 13:57:56.992591 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:56.992561    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/thanos-sidecar/0.log"
Oct 30 13:57:57.109832 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:57.109759    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/kube-rbac-proxy-web/0.log"
Oct 30 13:57:57.234481 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:57.234455    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/kube-rbac-proxy/0.log"
Oct 30 13:57:57.355632 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:57.355603    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/kube-rbac-proxy-thanos/0.log"
Oct 30 13:57:57.484258 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:57.484234    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c5a3e7ee-3d52-4005-aff1-f98799898c05/init-config-reloader/0.log"
Oct 30 13:57:57.962214 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:57.962137    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b454bf9f7-qlh67_74d25cd7-3283-4772-8267-c0f5d727633c/telemeter-client/0.log"
Oct 30 13:57:58.084314 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:58.084260    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b454bf9f7-qlh67_74d25cd7-3283-4772-8267-c0f5d727633c/reload/0.log"
Oct 30 13:57:58.199905 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:58.199874    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b454bf9f7-qlh67_74d25cd7-3283-4772-8267-c0f5d727633c/kube-rbac-proxy/0.log"
Oct 30 13:57:58.325283 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:58.325256    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff77c7974-6cgw8_3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/thanos-query/0.log"
Oct 30 13:57:58.439520 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:58.439462    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff77c7974-6cgw8_3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/kube-rbac-proxy-web/0.log"
Oct 30 13:57:58.550480 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:58.550457    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff77c7974-6cgw8_3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/kube-rbac-proxy/0.log"
Oct 30 13:57:58.662909 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:58.662830    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff77c7974-6cgw8_3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/prom-label-proxy/0.log"
Oct 30 13:57:58.772931 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:58.772903    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff77c7974-6cgw8_3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/kube-rbac-proxy-rules/0.log"
Oct 30 13:57:58.890073 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:57:58.890044    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5ff77c7974-6cgw8_3d4edeee-f5a5-497d-a53c-2a6c82a3fe85/kube-rbac-proxy-metrics/0.log"
Oct 30 13:58:00.562152 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:58:00.562125    5893 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cz4kh/perf-node-gather-daemonset-phhb4"
Oct 30 13:58:05.801828 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:58:05.801797    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rm7hq_6c8c3435-b6de-4aea-b596-970c9e17985c/dns/0.log"
Oct 30 13:58:05.918695 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:58:05.918669    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rm7hq_6c8c3435-b6de-4aea-b596-970c9e17985c/kube-rbac-proxy/0.log"
Oct 30 13:58:06.634536 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:58:06.634501    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tg5ns_52f5c091-40a2-46e8-9214-ddf02da33d0b/dns-node-resolver/0.log"
Oct 30 13:58:07.954237 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:58:07.954207    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bzkw4_dccd197b-ec88-445b-8de9-ff83832112bd/node-ca/0.log"
Oct 30 13:58:08.416785 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:58:08.416748    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-cbd4cb4f6-7pqlr_82007c50-3dfd-457b-9e0c-8d69f0481a9f/kube-auth-proxy/0.log"
Oct 30 13:58:08.890949 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:58:08.890922    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-jl8n2_742ed464-ac90-468a-92f7-edde51ba02bb/istio-proxy/0.log"
Oct 30 13:58:08.955689 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:58:08.955663    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-ktgn6_c66344b3-3175-40c0-b8a3-51ccf36bdc4f/istio-proxy/0.log"
Oct 30 13:58:09.078723 ip-10-0-141-232 kubenswrapper[5893]: I1030 13:58:09.078693    5893 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-ndplc_f49c9da9-a8bc-4139-bcc9-617e5a360704/istio-proxy/0.log"